[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:05 dist-test-slave-bc52 krb5kdc[11707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:05 dist-test-slave-bc52 krb5kdc[11707](info): set up 2 sockets
Jun 21 02:50:05 dist-test-slave-bc52 krb5kdc[11707](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 02:50:06.046243 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.015s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:06 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:06.211184 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37737
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:37737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:06.515726 11720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:06.516019 11720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:06.516141 11720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:06.523511 11720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:06.523645 11720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:06.523753 11720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:06.523854 11720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:06.523940 11720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:06.533520 11720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:37737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:06.534798 11720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:06.536074 11720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:06.548095 11725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:06.548048 11728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:06.548146 11726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:06.550565 11720 server_base.cc:1034] running on GCE node
I20260621 02:50:06.551265 11720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:06.552556 11720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:06.553756 11720 hybrid_clock.cc:648] HybridClock initialized: now 1782010206553689 us; error 52 us; skew 500 ppm
I20260621 02:50:06.554062 11720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:06 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010206, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:06.557030 11720 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:06.558458 11720 webserver.cc:460] Webserver started at http://127.11.109.62:42141/ using document root <none> and password file <none>
I20260621 02:50:06.559171 11720 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:06.559298 11720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:06.559595 11720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:06.562572 11720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "bc11506432bb4f349f71aa67d385b2b7"
format_stamp: "Formatted at 2026-06-21 02:50:06 on dist-test-slave-bc52"
server_key: "58d0214fb589cc75057ac68dff1f973e"
server_key_iv: "332d5e3df8439391e17253c85536cd33"
server_key_version: "encryptionkey@0"
I20260621 02:50:06.563330 11720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "bc11506432bb4f349f71aa67d385b2b7"
format_stamp: "Formatted at 2026-06-21 02:50:06 on dist-test-slave-bc52"
server_key: "58d0214fb589cc75057ac68dff1f973e"
server_key_iv: "332d5e3df8439391e17253c85536cd33"
server_key_version: "encryptionkey@0"
I20260621 02:50:06.568332 11720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260621 02:50:06.571681 11735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:06.572865 11720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:06.573035 11720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "bc11506432bb4f349f71aa67d385b2b7"
format_stamp: "Formatted at 2026-06-21 02:50:06 on dist-test-slave-bc52"
server_key: "58d0214fb589cc75057ac68dff1f973e"
server_key_iv: "332d5e3df8439391e17253c85536cd33"
server_key_version: "encryptionkey@0"
I20260621 02:50:06.573258 11720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:06.586771 11720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:06.590488 11720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:06.590760 11720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:06.608803 11720 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:37737
I20260621 02:50:06.608829 11786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:37737 every 8 connection(s)
I20260621 02:50:06.610201 11720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:06.614010 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:06.615563 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 11720
I20260621 02:50:06.615880 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:06.618268 11700 external_mini_cluster.cc:1428] Setting key 72fa0b659fa3e65f2f50eca7d535bd14
I20260621 02:50:06.626631 11787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7: Bootstrap starting.
I20260621 02:50:06.629536 11787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:06.630511 11787 log.cc:824] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:06.633780 11787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7: No bootstrap required, opened a new log
I20260621 02:50:06.642652 11787 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc11506432bb4f349f71aa67d385b2b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37737 } }
I20260621 02:50:06.643044 11787 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:06.643112 11787 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc11506432bb4f349f71aa67d385b2b7, State: Initialized, Role: FOLLOWER
I20260621 02:50:06.643718 11787 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc11506432bb4f349f71aa67d385b2b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37737 } }
I20260621 02:50:06.643954 11787 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:06.644093 11787 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:06.644277 11787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:06.646479 11787 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc11506432bb4f349f71aa67d385b2b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37737 } }
Jun 21 02:50:06 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:06.647105 11787 leader_election.cc:302] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc11506432bb4f349f71aa67d385b2b7; no voters:
I20260621 02:50:06.647766 11787 leader_election.cc:288] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:06.647878 11792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:06.649502 11792 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [term 1 LEADER]: Becoming Leader. State: Replica: bc11506432bb4f349f71aa67d385b2b7, State: Running, Role: LEADER
I20260621 02:50:06.650218 11787 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:06.650086 11792 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc11506432bb4f349f71aa67d385b2b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37737 } }
I20260621 02:50:06.653187 11793 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc11506432bb4f349f71aa67d385b2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc11506432bb4f349f71aa67d385b2b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37737 } } }
I20260621 02:50:06.653198 11794 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc11506432bb4f349f71aa67d385b2b7. Latest consensus state: current_term: 1 leader_uuid: "bc11506432bb4f349f71aa67d385b2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc11506432bb4f349f71aa67d385b2b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37737 } } }
I20260621 02:50:06.653623 11793 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:06.653685 11794 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:06.663489 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:06.622751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.11.109.62:37737)
0621 02:50:06.623653 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:06.623669 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:06.624946 (+ 1277us) server_negotiation.cc:407] Connection header received
0621 02:50:06.634209 (+ 9263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:06.634241 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:06.634690 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:06.635091 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:06.637659 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:06.639701 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:06.641401 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:06.642441 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:06.642620 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:06.649097 (+ 6477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:06.649115 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:06.649137 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:06.649221 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:06.654323 (+ 5102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:06.655685 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:06.655698 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:06.655709 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:06.655991 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:06.656697 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:06.656705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:06.656710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:06.657601 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:06.657984 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:06.661208 (+ 3224us) server_negotiation.cc:299] Negotiation successful
0621 02:50:06.661765 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":390,"threads_started":1}
I20260621 02:50:06.670249 11797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:06.675587 11797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:06.683900 11797 catalog_manager.cc:1347] Generated new cluster ID: 5b54da24be7b4bd09e8fbec766ee5dd4
I20260621 02:50:06.684027 11797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:06.695974 11797 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:06.697292 11797 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:06.706969 11797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc11506432bb4f349f71aa67d385b2b7: Generated new TSK 0
I20260621 02:50:06.707803 11797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:06.755165 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37737
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:07.073222 11813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:07.073531 11813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:07.073693 11813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:07.080730 11813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:07.080874 11813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:07.081043 11813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:07.090457 11813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37737
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:07.091687 11813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:07.093007 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:07.105957 11821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:07.106074 11819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:07.106102 11818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:07.106896 11813 server_base.cc:1034] running on GCE node
I20260621 02:50:07.107450 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:07.108316 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:07.109508 11813 hybrid_clock.cc:648] HybridClock initialized: now 1782010207109476 us; error 27 us; skew 500 ppm
I20260621 02:50:07.109822 11813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:07 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010207, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:07.112938 11813 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:07.114610 11813 webserver.cc:460] Webserver started at http://127.11.109.1:37379/ using document root <none> and password file <none>
I20260621 02:50:07.115288 11813 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:07.115422 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:07.115749 11813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:07.118928 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "2378742f952147d3b5dfc3c0f70c1a38"
format_stamp: "Formatted at 2026-06-21 02:50:07 on dist-test-slave-bc52"
server_key: "c85064ce62811b31a34fc422288b5a4a"
server_key_iv: "180a1247d390ccdacf19893c4c5db1f3"
server_key_version: "encryptionkey@0"
I20260621 02:50:07.119753 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "2378742f952147d3b5dfc3c0f70c1a38"
format_stamp: "Formatted at 2026-06-21 02:50:07 on dist-test-slave-bc52"
server_key: "c85064ce62811b31a34fc422288b5a4a"
server_key_iv: "180a1247d390ccdacf19893c4c5db1f3"
server_key_version: "encryptionkey@0"
I20260621 02:50:07.124958 11813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:50:07.128384 11828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:07.129710 11813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:07.129876 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "2378742f952147d3b5dfc3c0f70c1a38"
format_stamp: "Formatted at 2026-06-21 02:50:07 on dist-test-slave-bc52"
server_key: "c85064ce62811b31a34fc422288b5a4a"
server_key_iv: "180a1247d390ccdacf19893c4c5db1f3"
server_key_version: "encryptionkey@0"
I20260621 02:50:07.130060 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:07.143945 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:07.148111 11813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:07.148415 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:07.149365 11813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:07.150708 11813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:07.150779 11813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:07.150910 11813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:07.150979 11813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:07.185958 11813 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:33985
I20260621 02:50:07.186043 11940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:33985 every 8 connection(s)
I20260621 02:50:07.187186 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:07.192238 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 11813
I20260621 02:50:07.192468 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:07.193056 11700 external_mini_cluster.cc:1428] Setting key e27a4ee448ab311b8965ee0802a17060
Jun 21 02:50:07 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010207, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:50:07.208545 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:07.189665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:40451 (local address 127.11.109.62:37737)
0621 02:50:07.190081 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:07.190085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:07.191103 (+ 1018us) server_negotiation.cc:407] Connection header received
0621 02:50:07.192964 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:07.192969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:07.193059 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:07.193180 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:07.194797 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.195933 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.197665 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.198414 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.198557 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:07.202444 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:07.202449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:07.202452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:07.202482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:07.205122 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:07.205896 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:07.205901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:07.205913 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:07.206029 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:07.206552 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:07.206559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:07.206562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:07.206956 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:07.207150 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:07.207868 (+ 718us) server_negotiation.cc:299] Negotiation successful
0621 02:50:07.208166 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":221,"threads_started":1}
I20260621 02:50:07.209465 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:07.190271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37737 (local address 127.11.109.1:40451)
0621 02:50:07.190944 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:07.190981 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:07.192702 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:07.193441 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:07.193450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:07.194073 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:07.194605 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:07.194621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.196087 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.196090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:07.197522 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:07.197538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.198623 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.198632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:07.198922 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:07.199651 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:07.199672 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:07.202241 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:07.205329 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:07.205338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:07.205353 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:07.205770 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:07.206135 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:07.206140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:07.206144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:07.206398 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:07.207117 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:07.207126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:07.207493 (+ 367us) client_negotiation.cc:769] Sending connection context
0621 02:50:07.207699 (+ 206us) client_negotiation.cc:240] Negotiation successful
0621 02:50:07.208114 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":261,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:50:07.211112 11941 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37737
I20260621 02:50:07.211424 11941 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:07.212132 11941 heartbeater.cc:511] Master 127.11.109.62:37737 requested a full tablet report, sending...
I20260621 02:50:07.214325 11752 ts_manager.cc:194] Registered new tserver with Master: 2378742f952147d3b5dfc3c0f70c1a38 (127.11.109.1:33985)
I20260621 02:50:07.215724 11752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:40451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:07.230139 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37737
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:07.556996 11947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:07.557328 11947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:07.557512 11947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:07.564504 11947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:07.564657 11947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:07.564845 11947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:07.574510 11947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37737
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:07.575732 11947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:07.577081 11947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:07.589975 11952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:07.590020 11953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:07.589926 11955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:07.590958 11947 server_base.cc:1034] running on GCE node
I20260621 02:50:07.591545 11947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:07.592473 11947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:07.593705 11947 hybrid_clock.cc:648] HybridClock initialized: now 1782010207593653 us; error 43 us; skew 500 ppm
I20260621 02:50:07.594010 11947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:07 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010207, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:07.597141 11947 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:07.598642 11947 webserver.cc:460] Webserver started at http://127.11.109.2:46087/ using document root <none> and password file <none>
I20260621 02:50:07.599293 11947 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:07.599433 11947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:07.599736 11947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:07.602555 11947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "62088003833c42a4b21d20a6fb309657"
format_stamp: "Formatted at 2026-06-21 02:50:07 on dist-test-slave-bc52"
server_key: "6bf42dd9efe5e947222bc8fa90bfe22b"
server_key_iv: "f97b34ab4177c90cc8e7f3db1ef30990"
server_key_version: "encryptionkey@0"
I20260621 02:50:07.603287 11947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "62088003833c42a4b21d20a6fb309657"
format_stamp: "Formatted at 2026-06-21 02:50:07 on dist-test-slave-bc52"
server_key: "6bf42dd9efe5e947222bc8fa90bfe22b"
server_key_iv: "f97b34ab4177c90cc8e7f3db1ef30990"
server_key_version: "encryptionkey@0"
I20260621 02:50:07.608320 11947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:50:07.612429 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:07.613967 11947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:07.614120 11947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "62088003833c42a4b21d20a6fb309657"
format_stamp: "Formatted at 2026-06-21 02:50:07 on dist-test-slave-bc52"
server_key: "6bf42dd9efe5e947222bc8fa90bfe22b"
server_key_iv: "f97b34ab4177c90cc8e7f3db1ef30990"
server_key_version: "encryptionkey@0"
I20260621 02:50:07.614323 11947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:07.641739 11947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:07.645423 11947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:07.645658 11947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:07.646485 11947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:07.647634 11947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:07.647701 11947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:07.647778 11947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:07.647820 11947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:07.680224 11947 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:38103
I20260621 02:50:07.680284 12074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:38103 every 8 connection(s)
I20260621 02:50:07.681540 11947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:07.684216 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 11947
I20260621 02:50:07.684429 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:07.685007 11700 external_mini_cluster.cc:1428] Setting key 41de07f3c5cfc36d0801e2d0ba95c801
Jun 21 02:50:07 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010207, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:50:07.701253 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:07.683849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:47173 (local address 127.11.109.62:37737)
0621 02:50:07.684027 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:07.684031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:07.685427 (+ 1396us) server_negotiation.cc:407] Connection header received
0621 02:50:07.687478 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:07.687483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:07.687580 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:07.687696 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:07.689117 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.689914 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.691425 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.691826 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.691964 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:07.695499 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:07.695504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:07.695507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:07.695538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:07.698063 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:07.699024 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:07.699028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:07.699030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:07.699152 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:07.699623 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:07.699626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:07.699628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:07.699931 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:07.700043 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:07.700679 (+ 636us) server_negotiation.cc:299] Negotiation successful
0621 02:50:07.700976 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
I20260621 02:50:07.702301 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:07.684441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37737 (local address 127.11.109.2:47173)
0621 02:50:07.685278 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:07.685315 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:07.687224 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:07.687926 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:07.687935 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:07.688412 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:07.688945 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:07.688964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.690084 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.690091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:07.691270 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:07.691286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:07.692047 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:07.692054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:07.692274 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:07.692941 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:07.692961 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:07.695287 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:07.698255 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:07.698267 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:07.698284 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:07.698904 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:07.699268 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:07.699275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:07.699281 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:07.699521 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:07.700060 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:07.700067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:07.700382 (+ 315us) client_negotiation.cc:769] Sending connection context
0621 02:50:07.700609 (+ 227us) client_negotiation.cc:240] Negotiation successful
0621 02:50:07.701129 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":301,"threads_started":1}
I20260621 02:50:07.703758 12075 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37737
I20260621 02:50:07.704062 12075 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:07.704852 12075 heartbeater.cc:511] Master 127.11.109.62:37737 requested a full tablet report, sending...
I20260621 02:50:07.706429 11752 ts_manager.cc:194] Registered new tserver with Master: 62088003833c42a4b21d20a6fb309657 (127.11.109.2:38103)
I20260621 02:50:07.707710 11752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:47173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:07.724005 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37737
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:08.032263 12080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:08.032564 12080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:08.032771 12080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:08.039870 12080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:08.039989 12080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:08.040174 12080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:08.049497 12080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37737
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:08.050700 12080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:08.052100 12080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:08.064447 12086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:08.064479 12085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:08.064954 12088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:08.065155 12080 server_base.cc:1034] running on GCE node
I20260621 02:50:08.065755 12080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:08.066593 12080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:08.067811 12080 hybrid_clock.cc:648] HybridClock initialized: now 1782010208067755 us; error 53 us; skew 500 ppm
I20260621 02:50:08.068101 12080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:08.071085 12080 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:08.072443 12080 webserver.cc:460] Webserver started at http://127.11.109.3:44169/ using document root <none> and password file <none>
I20260621 02:50:08.073024 12080 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:08.073120 12080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:08.073419 12080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:08.076108 12080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "56d3e8c17d1142cc82d5599e1156dd76"
format_stamp: "Formatted at 2026-06-21 02:50:08 on dist-test-slave-bc52"
server_key: "d857b83f4f4d00590ea33ea040660cb9"
server_key_iv: "3c7c1f7e07a4c441f5026c62b3ecd9bf"
server_key_version: "encryptionkey@0"
I20260621 02:50:08.077349 12080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "56d3e8c17d1142cc82d5599e1156dd76"
format_stamp: "Formatted at 2026-06-21 02:50:08 on dist-test-slave-bc52"
server_key: "d857b83f4f4d00590ea33ea040660cb9"
server_key_iv: "3c7c1f7e07a4c441f5026c62b3ecd9bf"
server_key_version: "encryptionkey@0"
I20260621 02:50:08.082114 12080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:50:08.085098 12095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:08.086289 12080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:50:08.086455 12080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "56d3e8c17d1142cc82d5599e1156dd76"
format_stamp: "Formatted at 2026-06-21 02:50:08 on dist-test-slave-bc52"
server_key: "d857b83f4f4d00590ea33ea040660cb9"
server_key_iv: "3c7c1f7e07a4c441f5026c62b3ecd9bf"
server_key_version: "encryptionkey@0"
I20260621 02:50:08.086639 12080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:08.100257 12080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:08.104415 12080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:08.104691 12080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:08.105738 12080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:08.106971 12080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:08.107040 12080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:08.107203 12080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:08.107255 12080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:08.140053 12080 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:42117
I20260621 02:50:08.140097 12207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:42117 every 8 connection(s)
I20260621 02:50:08.141348 12080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:08.148311 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12080
I20260621 02:50:08.148518 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:08.149027 11700 external_mini_cluster.cc:1428] Setting key f27d921565672a732489148a6a4c2693
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:08.161054 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.143851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:34013 (local address 127.11.109.62:37737)
0621 02:50:08.144027 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:08.144031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:08.145250 (+ 1219us) server_negotiation.cc:407] Connection header received
0621 02:50:08.147165 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:08.147171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:08.147261 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:08.147376 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:08.149388 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.150228 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.152321 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.152888 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.153028 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.156222 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:08.156227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:08.156230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:08.156260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:08.158647 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.159276 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.159279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.159281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.159387 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.159732 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.159735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.159737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.159976 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:08.160078 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:08.160677 (+ 599us) server_negotiation.cc:299] Negotiation successful
0621 02:50:08.160825 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:50:08.162034 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.144366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37737 (local address 127.11.109.3:34013)
0621 02:50:08.145072 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:08.145114 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:08.146859 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:08.147615 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:08.147624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:08.148314 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:08.149140 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:08.149168 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.150423 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.150428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:08.152187 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:08.152200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.153094 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.153099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:08.153360 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.153979 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:08.154000 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:08.156032 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:08.158768 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:08.158774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:08.158787 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:08.159173 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:08.159472 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:08.159475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:08.159478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:08.159641 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:08.160074 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:08.160080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:08.160371 (+ 291us) client_negotiation.cc:769] Sending connection context
0621 02:50:08.160586 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 02:50:08.160937 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":296,"threads_started":1}
I20260621 02:50:08.163343 12208 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37737
I20260621 02:50:08.163671 12208 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:08.164533 12208 heartbeater.cc:511] Master 127.11.109.62:37737 requested a full tablet report, sending...
I20260621 02:50:08.165889 11752 ts_manager.cc:194] Registered new tserver with Master: 56d3e8c17d1142cc82d5599e1156dd76 (127.11.109.3:42117)
I20260621 02:50:08.166666 11752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:34013
I20260621 02:50:08.177443 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:08.219110 11941 heartbeater.cc:503] Master 127.11.109.62:37737 was elected leader, sending a full tablet report...
I20260621 02:50:08.225863 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.210654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.11.109.1:33985)
0621 02:50:08.211599 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:08.211606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:08.211631 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:50:08.211687 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:08.211692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:08.211893 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:08.212017 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:08.213401 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.214375 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.215530 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.216036 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.216124 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.220156 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:08.220166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:08.220179 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:08.220209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:08.223230 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.223857 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.223867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.223874 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.223999 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.224377 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.224383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.224386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.224719 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:08.225041 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:08.225294 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 02:50:08.225541 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":647,"threads_started":1}
W20260621 02:50:08.226979 11856 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34636
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:08.267587 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.254970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.11.109.1:33985)
0621 02:50:08.255182 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:08.255187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:08.255265 (+ 78us) server_negotiation.cc:407] Connection header received
0621 02:50:08.255403 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:08.255407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:08.255477 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:08.255556 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:08.256745 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.257510 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.258671 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.259104 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.259237 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.262907 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:08.262912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:08.262915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:08.262946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:08.265571 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.266097 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.266101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.266103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.266213 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.266512 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.266515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.266517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.266738 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:08.266901 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:08.267191 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 02:50:08.267354 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:50:08.269047 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 11813
I20260621 02:50:08.281435 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.270190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.11.109.1:33985)
0621 02:50:08.270368 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:08.270374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:08.270635 (+ 261us) server_negotiation.cc:407] Connection header received
0621 02:50:08.270806 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:08.270817 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:08.270869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:08.270970 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:08.272104 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.272772 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.274008 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.274410 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.274508 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.276542 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:08.276546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:08.276549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:08.276576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:08.279172 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.279761 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.279767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.279770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.279912 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.280216 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.280219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.280221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.280485 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:08.280649 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:08.280864 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 02:50:08.281007 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:50:08.282120 11936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:08.362782 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 11813
I20260621 02:50:08.378598 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 11947
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:08.393648 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.379478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.11.109.2:38103)
0621 02:50:08.379965 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:08.379972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:08.379996 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:50:08.380052 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:08.380058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:08.380272 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:08.380391 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:08.381582 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.382554 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.383753 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.384265 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.384366 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.388224 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:08.388233 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:08.388247 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:08.388276 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:08.391141 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.391762 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.391768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.391770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.391884 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.392228 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.392231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.392232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.392525 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:08.392748 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:08.393014 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 02:50:08.393234 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260621 02:50:08.394981 12070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:08.475239 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 11947
I20260621 02:50:08.490904 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12080
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[11707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:08.506271 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:08.491983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.11.109.3:42117)
0621 02:50:08.492636 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:08.492643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:08.492670 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:50:08.492725 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:08.492731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:08.492937 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:08.493056 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:08.494269 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.495675 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.496889 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:08.497484 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:08.497578 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:08.501398 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:08.501409 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:08.501423 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:08.501466 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:08.503990 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.504553 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.504559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.504562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.504669 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:08.504953 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:08.504956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:08.504958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:08.505321 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:08.505545 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:08.505783 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 02:50:08.505995 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":364,"threads_started":1}
I20260621 02:50:08.507465 12203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:08.588135 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12080
I20260621 02:50:08.603381 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 11720
I20260621 02:50:08.604535 11782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:08.704880 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 11720
2026-06-21T02:50:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3755 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[12229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[12229](info): set up 2 sockets
Jun 21 02:50:08 dist-test-slave-bc52 krb5kdc[12229](info): commencing operation
krb5kdc: starting...
W20260621 02:50:09.768898 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:09 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:09.936802 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46299
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:46299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:10.243561 12242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:10.243814 12242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:10.244000 12242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:10.251029 12242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:10.251140 12242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:10.251221 12242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:10.251318 12242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:10.251394 12242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:10.261096 12242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:46299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:10.262367 12242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:10.263638 12242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:10.275189 12250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:10.275156 12247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:10.275224 12248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:10.280037 12242 server_base.cc:1034] running on GCE node
I20260621 02:50:10.280792 12242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:10.282169 12242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:10.283358 12242 hybrid_clock.cc:648] HybridClock initialized: now 1782010210283305 us; error 51 us; skew 500 ppm
I20260621 02:50:10.283689 12242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:10 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010210, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:10.291370 12242 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:10.292822 12242 webserver.cc:460] Webserver started at http://127.11.109.62:41745/ using document root <none> and password file <none>
I20260621 02:50:10.293574 12242 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:10.293704 12242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:10.294024 12242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:10.296845 12242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c"
format_stamp: "Formatted at 2026-06-21 02:50:10 on dist-test-slave-bc52"
server_key: "29385b4b4d3123cdb720070b44e9ba36"
server_key_iv: "a8acc1ee100a3aebe148a9f5c09701ea"
server_key_version: "encryptionkey@0"
I20260621 02:50:10.297603 12242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c"
format_stamp: "Formatted at 2026-06-21 02:50:10 on dist-test-slave-bc52"
server_key: "29385b4b4d3123cdb720070b44e9ba36"
server_key_iv: "a8acc1ee100a3aebe148a9f5c09701ea"
server_key_version: "encryptionkey@0"
I20260621 02:50:10.302349 12242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 02:50:10.305488 12257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:10.306632 12242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:50:10.306790 12242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c"
format_stamp: "Formatted at 2026-06-21 02:50:10 on dist-test-slave-bc52"
server_key: "29385b4b4d3123cdb720070b44e9ba36"
server_key_iv: "a8acc1ee100a3aebe148a9f5c09701ea"
server_key_version: "encryptionkey@0"
I20260621 02:50:10.306973 12242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:10.330410 12242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:10.334407 12242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:10.334717 12242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:10.352711 12242 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:46299
I20260621 02:50:10.352748 12308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:46299 every 8 connection(s)
I20260621 02:50:10.354126 12242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:10.357770 12309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:10.360288 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12242
I20260621 02:50:10.360524 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:10.361217 11700 external_mini_cluster.cc:1428] Setting key 03127161671b09e79d0a2d216ec3901c
I20260621 02:50:10.369774 12309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c: Bootstrap starting.
I20260621 02:50:10.373445 12309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:10.374749 12309 log.cc:824] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:50:10 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:10.377882 12309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c: No bootstrap required, opened a new log
I20260621 02:50:10.384239 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:10.363233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.11.109.62:46299)
0621 02:50:10.364004 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:10.364016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:10.364054 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:50:10.365701 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:10.365722 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:10.366104 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:10.366411 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:10.368156 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:10.369506 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:10.371182 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:10.371851 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:10.371975 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:10.377023 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:10.377033 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:10.377048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:10.377080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:10.380229 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:10.381005 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:10.381011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:10.381017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:10.381244 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:10.381645 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:10.381648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:10.381650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:10.382130 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:10.382329 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:10.382870 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 02:50:10.383241 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":364,"threads_started":1}
I20260621 02:50:10.387101 12309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46299 } }
I20260621 02:50:10.387494 12309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:10.387598 12309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1874eb031b7a4cc889a1f2a2f8c0d53c, State: Initialized, Role: FOLLOWER
I20260621 02:50:10.388121 12309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46299 } }
I20260621 02:50:10.388341 12309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:10.388478 12309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:10.388646 12309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:10.390759 12309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46299 } }
I20260621 02:50:10.391227 12309 leader_election.cc:302] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1874eb031b7a4cc889a1f2a2f8c0d53c; no voters:
I20260621 02:50:10.391968 12309 leader_election.cc:288] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:10.392117 12314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:10.392793 12314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [term 1 LEADER]: Becoming Leader. State: Replica: 1874eb031b7a4cc889a1f2a2f8c0d53c, State: Running, Role: LEADER
I20260621 02:50:10.393373 12314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46299 } }
I20260621 02:50:10.393966 12309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:10.396278 12315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46299 } } }
I20260621 02:50:10.396317 12316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1874eb031b7a4cc889a1f2a2f8c0d53c. Latest consensus state: current_term: 1 leader_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1874eb031b7a4cc889a1f2a2f8c0d53c" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46299 } } }
I20260621 02:50:10.397325 12315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:10.397322 12316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:10.404531 12323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:10.409651 12323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:10.418102 12323 catalog_manager.cc:1347] Generated new cluster ID: fabae2fbee9342428624c99c93964150
I20260621 02:50:10.418207 12323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:10.431660 12323 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:10.432855 12323 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:10.443678 12323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1874eb031b7a4cc889a1f2a2f8c0d53c: Generated new TSK 0
I20260621 02:50:10.444453 12323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:10.491123 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46299
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:10.802016 12335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:10.802320 12335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:10.802489 12335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:10.809724 12335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:10.809846 12335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:10.810075 12335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:10.819568 12335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:10.820751 12335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:10.822053 12335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:10.834573 12340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:10.834630 12341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:10.834676 12343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:10.835361 12335 server_base.cc:1034] running on GCE node
I20260621 02:50:10.835908 12335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:10.836782 12335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:10.837992 12335 hybrid_clock.cc:648] HybridClock initialized: now 1782010210837957 us; error 51 us; skew 500 ppm
I20260621 02:50:10.838297 12335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:10 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010210, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:10.841248 12335 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:10.842655 12335 webserver.cc:460] Webserver started at http://127.11.109.1:44875/ using document root <none> and password file <none>
I20260621 02:50:10.843312 12335 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:10.843451 12335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:10.843755 12335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:10.846573 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed"
format_stamp: "Formatted at 2026-06-21 02:50:10 on dist-test-slave-bc52"
server_key: "11655c15c0e333ceb2a01579f6323cc8"
server_key_iv: "fe927ad3c327fd6aa2a2ba030a2cd729"
server_key_version: "encryptionkey@0"
I20260621 02:50:10.847325 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed"
format_stamp: "Formatted at 2026-06-21 02:50:10 on dist-test-slave-bc52"
server_key: "11655c15c0e333ceb2a01579f6323cc8"
server_key_iv: "fe927ad3c327fd6aa2a2ba030a2cd729"
server_key_version: "encryptionkey@0"
I20260621 02:50:10.851933 12335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:50:10.855060 12350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:10.856228 12335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:10.856384 12335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed"
format_stamp: "Formatted at 2026-06-21 02:50:10 on dist-test-slave-bc52"
server_key: "11655c15c0e333ceb2a01579f6323cc8"
server_key_iv: "fe927ad3c327fd6aa2a2ba030a2cd729"
server_key_version: "encryptionkey@0"
I20260621 02:50:10.856572 12335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:10.873361 12335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:10.876884 12335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:10.877161 12335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:10.878086 12335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:10.879302 12335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:10.879410 12335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:10.879561 12335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:10.879634 12335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:10.911664 12335 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:38983
I20260621 02:50:10.911675 12462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:38983 every 8 connection(s)
I20260621 02:50:10.912868 12335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:10.917812 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12335
I20260621 02:50:10.918025 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:10.918602 11700 external_mini_cluster.cc:1428] Setting key 3b4f763feac919e4988a3f53dc1816e2
Jun 21 02:50:10 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010210, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:10.934468 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:10.915294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:40943 (local address 127.11.109.62:46299)
0621 02:50:10.915706 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:10.915710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:10.916647 (+ 937us) server_negotiation.cc:407] Connection header received
0621 02:50:10.918565 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:10.918571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:10.918652 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:10.918774 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:10.920546 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:10.921328 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:10.923800 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:10.924185 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:10.924273 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:10.927761 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:10.927770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:10.927778 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:10.927822 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:10.931317 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:10.931982 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:10.931990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:10.931994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:10.932154 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:10.932534 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:10.932542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:10.932548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:10.932954 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:10.933127 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:10.933788 (+ 661us) server_negotiation.cc:299] Negotiation successful
0621 02:50:10.934038 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":212,"threads_started":1}
I20260621 02:50:10.935081 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:10.915860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46299 (local address 127.11.109.1:40943)
0621 02:50:10.916503 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:10.916539 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:10.918303 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:10.919052 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:10.919063 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:10.919733 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:10.920362 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:10.920381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:10.921476 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:10.921484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:10.923657 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:10.923674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:10.924398 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:10.924407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:10.924739 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:10.925546 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:10.925571 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:10.927564 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:10.931447 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:10.931454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:10.931465 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:10.931854 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:10.932265 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:10.932268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:10.932270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:10.932423 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:10.933093 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:10.933099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:10.933441 (+ 342us) client_negotiation.cc:769] Sending connection context
0621 02:50:10.933664 (+ 223us) client_negotiation.cc:240] Negotiation successful
0621 02:50:10.934003 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:50:10.936568 12463 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46299
I20260621 02:50:10.936867 12463 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:10.937618 12463 heartbeater.cc:511] Master 127.11.109.62:46299 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:50:10.939729 12274 ts_manager.cc:194] Registered new tserver with Master: f643ee1f30fa4b6eb9ae1594c3d854ed (127.11.109.1:38983)
I20260621 02:50:10.941699 12274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:40943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:10.958309 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46299
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:11.261062 12469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:11.261334 12469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:11.261430 12469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:11.268193 12469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:11.268282 12469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:11.268395 12469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:11.277734 12469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:11.278977 12469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:11.280203 12469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:11.292725 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:11.292749 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:11.292862 12477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:11.293385 12469 server_base.cc:1034] running on GCE node
I20260621 02:50:11.294062 12469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:11.294968 12469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:11.296181 12469 hybrid_clock.cc:648] HybridClock initialized: now 1782010211296139 us; error 25 us; skew 500 ppm
I20260621 02:50:11.296482 12469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:11 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010211, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:11.299441 12469 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:11.300757 12469 webserver.cc:460] Webserver started at http://127.11.109.2:42579/ using document root <none> and password file <none>
I20260621 02:50:11.301450 12469 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:11.301573 12469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:11.301869 12469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:11.304548 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "76fcd47bc0b940308750e9199fe42320"
format_stamp: "Formatted at 2026-06-21 02:50:11 on dist-test-slave-bc52"
server_key: "4c1b6fa6b67bf8281de517420a4f5faf"
server_key_iv: "f50c9c7401dff6386fe9eabd15e423be"
server_key_version: "encryptionkey@0"
I20260621 02:50:11.305387 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "76fcd47bc0b940308750e9199fe42320"
format_stamp: "Formatted at 2026-06-21 02:50:11 on dist-test-slave-bc52"
server_key: "4c1b6fa6b67bf8281de517420a4f5faf"
server_key_iv: "f50c9c7401dff6386fe9eabd15e423be"
server_key_version: "encryptionkey@0"
I20260621 02:50:11.310426 12469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:50:11.313647 12484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:11.314894 12469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:50:11.315075 12469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "76fcd47bc0b940308750e9199fe42320"
format_stamp: "Formatted at 2026-06-21 02:50:11 on dist-test-slave-bc52"
server_key: "4c1b6fa6b67bf8281de517420a4f5faf"
server_key_iv: "f50c9c7401dff6386fe9eabd15e423be"
server_key_version: "encryptionkey@0"
I20260621 02:50:11.315272 12469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:11.341360 12469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:11.345358 12469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:11.345680 12469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:11.346603 12469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:11.347915 12469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:11.348018 12469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:11.348157 12469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:11.348218 12469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:11.385134 12469 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:34813
I20260621 02:50:11.385196 12596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:34813 every 8 connection(s)
I20260621 02:50:11.386409 12469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:11.392239 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12469
I20260621 02:50:11.392462 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:11.392885 11700 external_mini_cluster.cc:1428] Setting key 6631458c9c51d20237cf3d6820657585
Jun 21 02:50:11 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010211, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:11.408753 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.388853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:53751 (local address 127.11.109.62:46299)
0621 02:50:11.389067 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:11.389074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:11.390221 (+ 1147us) server_negotiation.cc:407] Connection header received
0621 02:50:11.391951 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:11.391959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:11.392055 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:11.392178 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:11.393781 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.394814 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.397407 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.397807 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.397902 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.401745 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:11.401754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:11.401762 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:11.401807 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:11.405564 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:11.406387 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:11.406395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:11.406400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:11.406581 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:11.406982 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:11.406988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:11.406991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:11.407326 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:11.407443 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:11.408360 (+ 917us) server_negotiation.cc:299] Negotiation successful
0621 02:50:11.408527 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:50:11.410317 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.389419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46299 (local address 127.11.109.2:53751)
0621 02:50:11.390064 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:11.390103 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:11.391680 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:11.392418 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:11.392426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:11.393001 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:11.393598 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.393614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.394994 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.395000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.397264 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.397289 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.398022 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.398030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.398347 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.399060 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:11.399094 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:11.401522 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:11.405711 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:11.405717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:11.405730 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:11.406262 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:11.406691 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:11.406695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:11.406697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:11.406860 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:11.407462 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:11.407474 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:11.407951 (+ 477us) client_negotiation.cc:769] Sending connection context
0621 02:50:11.408294 (+ 343us) client_negotiation.cc:240] Negotiation successful
0621 02:50:11.408863 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":249,"threads_started":1}
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:50:11.411863 12597 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46299
I20260621 02:50:11.412173 12597 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:11.412866 12597 heartbeater.cc:511] Master 127.11.109.62:46299 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
I20260621 02:50:11.414710 12274 ts_manager.cc:194] Registered new tserver with Master: 76fcd47bc0b940308750e9199fe42320 (127.11.109.2:34813)
I20260621 02:50:11.415512 12274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:53751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:11.435878 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46299
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:11.740782 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:11.741050 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:11.741225 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:11.748211 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:11.748304 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:11.748464 12602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:11.757864 12602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:11.759218 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:11.761063 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:11.773861 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:11.773933 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:11.773967 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:11.774703 12602 server_base.cc:1034] running on GCE node
I20260621 02:50:11.775238 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:11.776075 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:11.777276 12602 hybrid_clock.cc:648] HybridClock initialized: now 1782010211777223 us; error 39 us; skew 500 ppm
I20260621 02:50:11.777594 12602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:11 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010211, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:11.781026 12602 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:11.784953 12602 webserver.cc:460] Webserver started at http://127.11.109.3:44245/ using document root <none> and password file <none>
I20260621 02:50:11.785810 12602 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:11.785940 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:11.786317 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:11.790071 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "0fe3c5ef434340f9878c6990d22be930"
format_stamp: "Formatted at 2026-06-21 02:50:11 on dist-test-slave-bc52"
server_key: "d9da3f63ce96b732aad94b975b206cc2"
server_key_iv: "649ee39038d79b61db543b63af12a943"
server_key_version: "encryptionkey@0"
I20260621 02:50:11.790827 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "0fe3c5ef434340f9878c6990d22be930"
format_stamp: "Formatted at 2026-06-21 02:50:11 on dist-test-slave-bc52"
server_key: "d9da3f63ce96b732aad94b975b206cc2"
server_key_iv: "649ee39038d79b61db543b63af12a943"
server_key_version: "encryptionkey@0"
I20260621 02:50:11.795758 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:50:11.799115 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:11.800472 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:11.800629 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "0fe3c5ef434340f9878c6990d22be930"
format_stamp: "Formatted at 2026-06-21 02:50:11 on dist-test-slave-bc52"
server_key: "d9da3f63ce96b732aad94b975b206cc2"
server_key_iv: "649ee39038d79b61db543b63af12a943"
server_key_version: "encryptionkey@0"
I20260621 02:50:11.800827 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:11.817696 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:11.822079 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:11.822355 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:11.823261 12602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:11.824517 12602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:11.824601 12602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:11.824787 12602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:11.824838 12602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:11.855597 12602 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:41983
I20260621 02:50:11.855633 12729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:41983 every 8 connection(s)
I20260621 02:50:11.856879 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:11.860985 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12602
I20260621 02:50:11.861220 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:11.861660 11700 external_mini_cluster.cc:1428] Setting key f3f01549e4bc9d1880f361bd710a46e8
Jun 21 02:50:11 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010211, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:11.876448 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.859351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:60767 (local address 127.11.109.62:46299)
0621 02:50:11.859525 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:11.859530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:11.860866 (+ 1336us) server_negotiation.cc:407] Connection header received
0621 02:50:11.863098 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:11.863103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:11.863195 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:11.863316 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:11.864979 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.865880 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.867325 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.867703 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.867790 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.870924 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:11.870928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:11.870931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:11.870961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:11.873442 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:11.874040 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:11.874044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:11.874046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:11.874154 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:11.874531 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:11.874538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:11.874540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:11.874918 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:11.875095 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:11.875889 (+ 794us) server_negotiation.cc:299] Negotiation successful
0621 02:50:11.876105 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:11.877465 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.859856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46299 (local address 127.11.109.3:60767)
0621 02:50:11.860686 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:11.860740 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:11.862774 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:11.863611 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:11.863626 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:11.864119 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:11.864830 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.864847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.866012 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.866016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.867216 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.867226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.867868 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.867872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.868076 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.868651 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:11.868670 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:11.870737 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:11.873557 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:11.873563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:11.873574 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:11.873948 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:11.874242 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:11.874244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:11.874247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:11.874410 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:11.875080 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:11.875088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:11.875520 (+ 432us) client_negotiation.cc:769] Sending connection context
0621 02:50:11.875823 (+ 303us) client_negotiation.cc:240] Negotiation successful
0621 02:50:11.876309 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":348,"threads_started":1}
I20260621 02:50:11.878711 12730 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46299
I20260621 02:50:11.878954 12730 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:11.879591 12730 heartbeater.cc:511] Master 127.11.109.62:46299 requested a full tablet report, sending...
I20260621 02:50:11.880923 12274 ts_manager.cc:194] Registered new tserver with Master: 0fe3c5ef434340f9878c6990d22be930 (127.11.109.3:41983)
I20260621 02:50:11.881762 12274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:60767
I20260621 02:50:11.888906 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:50:11.906868 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.894844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.11.109.62:46299)
0621 02:50:11.894994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:11.894998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:11.895462 (+ 464us) server_negotiation.cc:407] Connection header received
0621 02:50:11.895657 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:11.895662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:11.895727 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:11.895826 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:11.897219 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.897929 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.899178 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.899598 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.899723 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.901999 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:11.902004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:11.902013 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:11.902041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:11.904243 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:11.904896 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:11.904904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:11.904908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:11.905038 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:11.905445 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:11.905452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:11.905455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:11.905772 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:11.906020 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:11.906372 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 02:50:11.906581 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:50:11.913151 12274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:11.919796 12274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:11.943504 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.930843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:34813 (local address 127.0.0.1:55548)
0621 02:50:11.933451 (+ 2608us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:11.933489 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:11.933654 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:11.934443 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:11.934453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:11.934502 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:11.934947 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.934962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.937517 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.937525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.940377 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.940395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.942556 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.942562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.942753 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.942807 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 02:50:11.942934 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:50:11.943133 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387,"thread_start_us":2211,"threads_started":1}
I20260621 02:50:11.943751 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.930934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.11.109.2:34813)
0621 02:50:11.932032 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:11.932038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:11.933857 (+ 1819us) server_negotiation.cc:407] Connection header received
0621 02:50:11.933960 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:11.933969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:11.934212 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:11.935165 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:11.935437 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.937345 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.940617 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.942351 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.942542 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.942786 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:11.943273 (+ 487us) server_negotiation.cc:299] Negotiation successful
0621 02:50:11.943432 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":834,"threads_started":1}
I20260621 02:50:11.945148 12463 heartbeater.cc:503] Master 127.11.109.62:46299 was elected leader, sending a full tablet report...
I20260621 02:50:11.948324 12532 tablet_service.cc:1467] Processing CreateTablet for tablet 586af1b5e8be4bd6a805cfcb63f23725 (DEFAULT_TABLE table=test-table [id=9dbd32a1da3b49f587f1db1874968c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:11.949723 12532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586af1b5e8be4bd6a805cfcb63f23725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:11.951092 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.930918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:38983 (local address 127.0.0.1:57668)
0621 02:50:11.941525 (+ 10607us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:11.941547 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:11.941676 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:11.943296 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:11.943301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:11.943326 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:11.944412 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.944425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.946242 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.946250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.948549 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.948560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.950261 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.950269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.950427 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.950458 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:50:11.950564 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:50:11.950700 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10452,"mutex_wait_us":62,"thread_start_us":10315,"threads_started":1}
I20260621 02:50:11.951117 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.931429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.11.109.1:38983)
0621 02:50:11.942523 (+ 11094us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:11.942534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:11.942559 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:50:11.942654 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:11.942662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:11.942881 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:11.943075 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:11.944652 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.946123 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.948683 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.950054 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.950201 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.950416 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:11.950616 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 02:50:11.950784 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10921,"thread_start_us":10797,"threads_started":1}
I20260621 02:50:11.951778 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.933646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:41983 (local address 127.0.0.1:38918)
0621 02:50:11.941812 (+ 8166us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:11.941832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:11.941941 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:11.943928 (+ 1987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:11.943936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:11.943967 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:11.944419 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.944426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.947152 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.947160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.949236 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:11.949249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.950985 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.950992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:11.951189 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.951230 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 02:50:11.951297 (+ 67us) client_negotiation.cc:240] Negotiation successful
0621 02:50:11.951406 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8033,"mutex_wait_us":1,"thread_start_us":7899,"threads_started":1}
I20260621 02:50:11.953415 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:11.934202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.11.109.3:41983)
0621 02:50:11.943301 (+ 9099us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:11.943309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:11.943336 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:50:11.943414 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:11.943420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:11.943617 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:11.943870 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:11.945408 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.946981 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.949367 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:11.950756 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:11.952792 (+ 2036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:11.952949 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:11.953007 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 02:50:11.953108 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8951,"thread_start_us":8804,"threads_started":1}
I20260621 02:50:11.955367 12398 tablet_service.cc:1467] Processing CreateTablet for tablet 586af1b5e8be4bd6a805cfcb63f23725 (DEFAULT_TABLE table=test-table [id=9dbd32a1da3b49f587f1db1874968c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:11.956086 12665 tablet_service.cc:1467] Processing CreateTablet for tablet 586af1b5e8be4bd6a805cfcb63f23725 (DEFAULT_TABLE table=test-table [id=9dbd32a1da3b49f587f1db1874968c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:11.956779 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586af1b5e8be4bd6a805cfcb63f23725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:11.957281 12665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586af1b5e8be4bd6a805cfcb63f23725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:11.964159 12748 tablet_bootstrap.cc:492] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: Bootstrap starting.
I20260621 02:50:11.967710 12748 tablet_bootstrap.cc:654] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:11.969095 12748 log.cc:824] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:11.969344 12749 tablet_bootstrap.cc:492] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: Bootstrap starting.
I20260621 02:50:11.972254 12749 tablet_bootstrap.cc:654] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:11.973256 12749 log.cc:824] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:11.975872 12749 tablet_bootstrap.cc:492] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: No bootstrap required, opened a new log
I20260621 02:50:11.976130 12749 ts_tablet_manager.cc:1397] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260621 02:50:11.977102 12748 tablet_bootstrap.cc:492] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: No bootstrap required, opened a new log
I20260621 02:50:11.977396 12748 ts_tablet_manager.cc:1397] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260621 02:50:11.977414 12750 tablet_bootstrap.cc:492] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: Bootstrap starting.
I20260621 02:50:11.980774 12750 tablet_bootstrap.cc:654] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:11.981997 12750 log.cc:824] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:11.984030 12749 raft_consensus.cc:348] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:11.984375 12749 raft_consensus.cc:374] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:11.984460 12749 raft_consensus.cc:729] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f643ee1f30fa4b6eb9ae1594c3d854ed, State: Initialized, Role: FOLLOWER
I20260621 02:50:11.985005 12750 tablet_bootstrap.cc:492] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: No bootstrap required, opened a new log
I20260621 02:50:11.985034 12749 consensus_queue.cc:260] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:11.985298 12750 ts_tablet_manager.cc:1397] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260621 02:50:11.986596 12749 ts_tablet_manager.cc:1428] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260621 02:50:11.987305 12748 raft_consensus.cc:348] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:11.987712 12748 raft_consensus.cc:374] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:11.987821 12748 raft_consensus.cc:729] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fcd47bc0b940308750e9199fe42320, State: Initialized, Role: FOLLOWER
I20260621 02:50:11.988368 12748 consensus_queue.cc:260] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:11.990669 12597 heartbeater.cc:503] Master 127.11.109.62:46299 was elected leader, sending a full tablet report...
I20260621 02:50:11.991312 12748 ts_tablet_manager.cc:1428] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260621 02:50:11.995482 12750 raft_consensus.cc:348] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:11.995814 12750 raft_consensus.cc:374] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:11.995934 12750 raft_consensus.cc:729] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe3c5ef434340f9878c6990d22be930, State: Initialized, Role: FOLLOWER
I20260621 02:50:11.996438 12750 consensus_queue.cc:260] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:11.997366 12730 heartbeater.cc:503] Master 127.11.109.62:46299 was elected leader, sending a full tablet report...
I20260621 02:50:11.997934 12750 ts_tablet_manager.cc:1428] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260621 02:50:12.021929 12756 raft_consensus.cc:482] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:12.022171 12756 raft_consensus.cc:504] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:12.023685 12756 leader_election.cc:288] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76fcd47bc0b940308750e9199fe42320 (127.11.109.2:34813), f643ee1f30fa4b6eb9ae1594c3d854ed (127.11.109.1:38983)
I20260621 02:50:12.029687 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.024101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:53889 (local address 127.11.109.2:34813)
0621 02:50:12.024283 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.024290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.024310 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:50:12.024390 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.024397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.024485 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.024594 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:12.025017 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.026018 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.027817 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.028886 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.028996 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.029055 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.029406 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.029497 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:50:12.029816 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.023918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:34813 (local address 127.11.109.3:53889)
0621 02:50:12.024095 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:12.024115 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:12.024239 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:12.024627 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:12.024642 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:12.024668 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:12.024835 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:12.024843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.026147 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.026152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:12.027673 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:12.027684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.029041 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.029047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:12.029250 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.029288 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:50:12.029386 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:50:12.029531 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":6}
I20260621 02:50:12.030751 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.024095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:48727 (local address 127.11.109.1:38983)
0621 02:50:12.024283 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.024289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.024905 (+ 616us) server_negotiation.cc:407] Connection header received
0621 02:50:12.025045 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.025048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.025105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.025228 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:12.025773 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.026723 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.028666 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.029784 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.029940 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.030015 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.030367 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.030529 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:50:12.030766 12552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586af1b5e8be4bd6a805cfcb63f23725" candidate_uuid: "0fe3c5ef434340f9878c6990d22be930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fcd47bc0b940308750e9199fe42320" is_pre_election: true
I20260621 02:50:12.031165 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.023931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:38983 (local address 127.11.109.3:48727)
0621 02:50:12.024798 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:12.024811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:12.024904 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:12.025250 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:12.025254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:12.025279 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:12.025627 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:12.025637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.026865 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.026868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:12.028497 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:12.028520 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.030068 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.030074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:12.030208 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.030229 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:50:12.030332 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:50:12.030796 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":124,"thread_start_us":398,"threads_started":1}
I20260621 02:50:12.031358 12552 raft_consensus.cc:2454] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe3c5ef434340f9878c6990d22be930 in term 0.
I20260621 02:50:12.032096 12619 leader_election.cc:302] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe3c5ef434340f9878c6990d22be930, 76fcd47bc0b940308750e9199fe42320; no voters:
I20260621 02:50:12.032155 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586af1b5e8be4bd6a805cfcb63f23725" candidate_uuid: "0fe3c5ef434340f9878c6990d22be930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" is_pre_election: true
I20260621 02:50:12.032514 12756 raft_consensus.cc:2781] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:12.032618 12756 raft_consensus.cc:482] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:12.032680 12756 raft_consensus.cc:3037] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:12.032666 12418 raft_consensus.cc:2454] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe3c5ef434340f9878c6990d22be930 in term 0.
I20260621 02:50:12.034852 12756 raft_consensus.cc:504] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:12.035430 12756 leader_election.cc:288] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [CANDIDATE]: Term 1 election: Requested vote from peers 76fcd47bc0b940308750e9199fe42320 (127.11.109.2:34813), f643ee1f30fa4b6eb9ae1594c3d854ed (127.11.109.1:38983)
I20260621 02:50:12.035852 12552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586af1b5e8be4bd6a805cfcb63f23725" candidate_uuid: "0fe3c5ef434340f9878c6990d22be930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fcd47bc0b940308750e9199fe42320"
I20260621 02:50:12.035881 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586af1b5e8be4bd6a805cfcb63f23725" candidate_uuid: "0fe3c5ef434340f9878c6990d22be930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed"
I20260621 02:50:12.036052 12552 raft_consensus.cc:3037] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:12.036055 12418 raft_consensus.cc:3037] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:12.038633 12418 raft_consensus.cc:2454] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe3c5ef434340f9878c6990d22be930 in term 1.
I20260621 02:50:12.038652 12552 raft_consensus.cc:2454] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe3c5ef434340f9878c6990d22be930 in term 1.
I20260621 02:50:12.039017 12618 leader_election.cc:302] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe3c5ef434340f9878c6990d22be930, f643ee1f30fa4b6eb9ae1594c3d854ed; no voters:
I20260621 02:50:12.039372 12756 raft_consensus.cc:2781] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:12.039839 12756 raft_consensus.cc:686] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [term 1 LEADER]: Becoming Leader. State: Replica: 0fe3c5ef434340f9878c6990d22be930, State: Running, Role: LEADER
I20260621 02:50:12.040343 12756 consensus_queue.cc:237] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } }
I20260621 02:50:12.044596 12273 catalog_manager.cc:5495] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe3c5ef434340f9878c6990d22be930 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "0fe3c5ef434340f9878c6990d22be930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fcd47bc0b940308750e9199fe42320" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f643ee1f30fa4b6eb9ae1594c3d854ed" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 38983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe3c5ef434340f9878c6990d22be930" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41983 } health_report { overall_health: HEALTHY } } }
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:12.077328 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.063968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.11.109.1:38983)
0621 02:50:12.064184 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.064194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.064576 (+ 382us) server_negotiation.cc:407] Connection header received
0621 02:50:12.064705 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.064709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.064767 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.064846 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:12.066309 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.067041 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.068182 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.068578 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.068666 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.072287 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:12.072299 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:12.072314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:12.072344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:12.075133 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.075698 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.075704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.075706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.075818 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.076105 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.076108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.076110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.076415 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:12.076637 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.076805 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.076944 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260621 02:50:12.108547 12731 tablet.cc:2369] T 586af1b5e8be4bd6a805cfcb63f23725 P 0fe3c5ef434340f9878c6990d22be930: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:12.121135 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.107879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.11.109.1:38983)
0621 02:50:12.108113 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.108121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.108472 (+ 351us) server_negotiation.cc:407] Connection header received
0621 02:50:12.108670 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.108679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.108760 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.108841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:12.110335 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.111139 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.112324 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.112719 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.112805 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.116259 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:12.116263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:12.116267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:12.116297 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:12.119083 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.119678 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.119682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.119684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.119806 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.120131 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.120134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.120136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.120373 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:12.120557 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.120755 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.120902 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 02:50:12.138299 12598 tablet.cc:2369] T 586af1b5e8be4bd6a805cfcb63f23725 P 76fcd47bc0b940308750e9199fe42320: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:12.161429 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.148762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.11.109.1:38983)
0621 02:50:12.149016 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.149023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.149302 (+ 279us) server_negotiation.cc:407] Connection header received
0621 02:50:12.149505 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.149515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.149575 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.149677 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:12.150936 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.151620 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.152793 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.153267 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.153388 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.157024 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:12.157029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:12.157033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:12.157068 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:12.159348 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.159907 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.159911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.159913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.160022 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.160314 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.160317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.160319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.160575 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:12.160753 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.160966 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.161127 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 02:50:12.163065 12378 tablet_service.cc:2908] Rejecting scan request for tablet 586af1b5e8be4bd6a805cfcb63f23725: Uninitialized: clean time has not yet been initialized
W20260621 02:50:12.165221 12464 tablet.cc:2369] T 586af1b5e8be4bd6a805cfcb63f23725 P f643ee1f30fa4b6eb9ae1594c3d854ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:50:12.165863 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12335
I20260621 02:50:12.176827 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.166641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.11.109.1:38983)
0621 02:50:12.166832 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.166838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.167117 (+ 279us) server_negotiation.cc:407] Connection header received
0621 02:50:12.167225 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.167229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.167289 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.167359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:12.168418 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.169152 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.170306 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.170675 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.170771 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.172670 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:12.172674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:12.172677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:12.172699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:12.174842 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.175389 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.175392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.175394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.175491 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.175761 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.175764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.175766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.175991 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:12.176186 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.176376 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.176515 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:50:12.177385 12458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:12.266139 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12335
I20260621 02:50:12.282287 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12469
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:12.295722 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.283070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.11.109.2:34813)
0621 02:50:12.283256 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.283261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.283278 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:50:12.283412 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.283416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.283476 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.283559 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:12.284650 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.285405 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.286552 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.286917 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.286999 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.290793 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:12.290807 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:12.290821 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:12.290854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:12.293538 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.294094 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.294113 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.294115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.294231 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.294599 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.294602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.294604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.294922 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:12.295145 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.295344 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.295487 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:50:12.296381 12592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:12.380983 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12469
I20260621 02:50:12.396938 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12602
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:12.411365 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:12.397739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.11.109.3:41983)
0621 02:50:12.397920 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:12.397924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:12.397940 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:50:12.398064 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:12.398068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:12.398123 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:12.398208 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:12.399289 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.400001 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.401294 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:12.401701 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:12.401792 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:12.406026 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:12.406040 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:12.406057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:12.406086 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:12.408667 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.409341 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.409349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.409351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.409482 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:12.409857 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:12.409864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:12.409867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:12.410260 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:12.410545 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:12.410825 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 02:50:12.411066 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:50:12.411955 12725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:12.497947 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12602
I20260621 02:50:12.510818 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12242
I20260621 02:50:12.511993 12304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:12.617152 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12242
2026-06-21T02:50:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3911 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12777](info): set up 2 sockets
Jun 21 02:50:12 dist-test-slave-bc52 krb5kdc[12777](info): commencing operation
krb5kdc: starting...
W20260621 02:50:13.692981 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:13 dist-test-slave-bc52 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:13.860183 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:34177
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:34177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:14.165776 12790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:14.166045 12790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:14.166208 12790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:14.173183 12790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:14.173341 12790 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:50:14.173430 12790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:14.173547 12790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:14.173630 12790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:14.173717 12790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:14.183871 12790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:34177
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:34177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:14.185137 12790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:14.186442 12790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:14.197582 12798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:14.197621 12796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:14.197656 12795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:14.198333 12790 server_base.cc:1034] running on GCE node
I20260621 02:50:14.199004 12790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:14.200217 12790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:14.201422 12790 hybrid_clock.cc:648] HybridClock initialized: now 1782010214201369 us; error 32 us; skew 500 ppm
I20260621 02:50:14.201728 12790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:14 dist-test-slave-bc52 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:14.205065 12790 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:14.206892 12790 webserver.cc:460] Webserver started at http://127.11.109.62:40593/ using document root <none> and password file <none>
I20260621 02:50:14.207710 12790 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:14.207856 12790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:14.208212 12790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:14.211916 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "9090a2620e3e4f69b3e15bac50780a50"
format_stamp: "Formatted at 2026-06-21 02:50:14 on dist-test-slave-bc52"
server_key: "8dd9bc5b204f89e43329272c1ef2eb39"
server_key_iv: "e7d5cd9cefb523eee1721236bd8b5fd3"
server_key_version: "encryptionkey@0"
I20260621 02:50:14.212890 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "9090a2620e3e4f69b3e15bac50780a50"
format_stamp: "Formatted at 2026-06-21 02:50:14 on dist-test-slave-bc52"
server_key: "8dd9bc5b204f89e43329272c1ef2eb39"
server_key_iv: "e7d5cd9cefb523eee1721236bd8b5fd3"
server_key_version: "encryptionkey@0"
I20260621 02:50:14.218220 12790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 02:50:14.221365 12805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:14.222561 12790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:50:14.222724 12790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "9090a2620e3e4f69b3e15bac50780a50"
format_stamp: "Formatted at 2026-06-21 02:50:14 on dist-test-slave-bc52"
server_key: "8dd9bc5b204f89e43329272c1ef2eb39"
server_key_iv: "e7d5cd9cefb523eee1721236bd8b5fd3"
server_key_version: "encryptionkey@0"
I20260621 02:50:14.222923 12790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:14.253801 12790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:14.257997 12790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:14.258266 12790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:14.278869 12790 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:34177
I20260621 02:50:14.278888 12866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:34177 every 8 connection(s)
I20260621 02:50:14.280208 12790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:14.284060 12867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:14.285156 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12790
I20260621 02:50:14.285416 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:14.286067 11700 external_mini_cluster.cc:1428] Setting key a7f396710a65a3ce19030d0634d8c113
I20260621 02:50:14.297535 12867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: Bootstrap starting.
Jun 21 02:50:14 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:14.301158 12867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:14.302372 12867 log.cc:824] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:14.305327 12867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: No bootstrap required, opened a new log
I20260621 02:50:14.308856 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:14.287976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.11.109.62:34177)
0621 02:50:14.288773 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:14.288783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:14.288823 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:50:14.290429 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:14.290454 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:14.290817 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:14.291125 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:14.292724 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.294084 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.295826 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.296430 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.296552 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:14.301210 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:14.301224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:14.301247 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:14.301287 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:14.305214 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:14.305873 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:14.305881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:14.305888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:14.306080 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:14.306422 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:14.306428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:14.306431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:14.306905 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:14.307199 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:14.307583 (+ 384us) server_negotiation.cc:299] Negotiation successful
0621 02:50:14.307944 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":389,"threads_started":1}
I20260621 02:50:14.312815 12867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9090a2620e3e4f69b3e15bac50780a50" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34177 } }
I20260621 02:50:14.313140 12867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:14.313295 12867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9090a2620e3e4f69b3e15bac50780a50, State: Initialized, Role: FOLLOWER
I20260621 02:50:14.313844 12867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9090a2620e3e4f69b3e15bac50780a50" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34177 } }
I20260621 02:50:14.314105 12867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:14.314275 12867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:14.314460 12867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:14.316423 12867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9090a2620e3e4f69b3e15bac50780a50" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34177 } }
I20260621 02:50:14.316866 12867 leader_election.cc:302] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9090a2620e3e4f69b3e15bac50780a50; no voters:
I20260621 02:50:14.317483 12867 leader_election.cc:288] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:14.317580 12872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:14.318234 12872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [term 1 LEADER]: Becoming Leader. State: Replica: 9090a2620e3e4f69b3e15bac50780a50, State: Running, Role: LEADER
I20260621 02:50:14.318747 12872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9090a2620e3e4f69b3e15bac50780a50" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34177 } }
I20260621 02:50:14.319485 12867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:14.321477 12874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9090a2620e3e4f69b3e15bac50780a50. Latest consensus state: current_term: 1 leader_uuid: "9090a2620e3e4f69b3e15bac50780a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9090a2620e3e4f69b3e15bac50780a50" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34177 } } }
I20260621 02:50:14.321954 12874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:14.321586 12873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9090a2620e3e4f69b3e15bac50780a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9090a2620e3e4f69b3e15bac50780a50" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34177 } } }
I20260621 02:50:14.323221 12873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:14.323691 12877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:14.328578 12877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:14.337517 12877 catalog_manager.cc:1347] Generated new cluster ID: 420f7e5f966e487e9bfa5ab4379f739c
I20260621 02:50:14.337615 12877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:14.363087 12877 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:14.364327 12877 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:14.372236 12877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: Generated new TSK 0
I20260621 02:50:14.373001 12877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:14.426384 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34177
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:14.744436 12893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:14.744693 12893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:14.744789 12893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:14.751811 12893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:14.751911 12893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:14.752070 12893 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:50:14.752158 12893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:14.762112 12893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:34177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:14.763306 12893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:14.764586 12893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:14.777148 12901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:14.777113 12898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:14.777212 12899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:14.778369 12893 server_base.cc:1034] running on GCE node
I20260621 02:50:14.778970 12893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:14.779906 12893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:14.781126 12893 hybrid_clock.cc:648] HybridClock initialized: now 1782010214781075 us; error 39 us; skew 500 ppm
I20260621 02:50:14.781447 12893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:14 dist-test-slave-bc52 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:14.784461 12893 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:14.786034 12893 webserver.cc:460] Webserver started at http://127.11.109.1:43033/ using document root <none> and password file <none>
I20260621 02:50:14.786721 12893 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:14.786835 12893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:14.787168 12893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:14.790057 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "b7cff584bd8f43b896dc47eebde0d997"
format_stamp: "Formatted at 2026-06-21 02:50:14 on dist-test-slave-bc52"
server_key: "a97126e1cb7a988347e68126c7388957"
server_key_iv: "bc1eb3054ba423dfbb4ac94c92ea5e18"
server_key_version: "encryptionkey@0"
I20260621 02:50:14.790833 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "b7cff584bd8f43b896dc47eebde0d997"
format_stamp: "Formatted at 2026-06-21 02:50:14 on dist-test-slave-bc52"
server_key: "a97126e1cb7a988347e68126c7388957"
server_key_iv: "bc1eb3054ba423dfbb4ac94c92ea5e18"
server_key_version: "encryptionkey@0"
I20260621 02:50:14.795888 12893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:50:14.799237 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:14.800503 12893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:14.800664 12893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "b7cff584bd8f43b896dc47eebde0d997"
format_stamp: "Formatted at 2026-06-21 02:50:14 on dist-test-slave-bc52"
server_key: "a97126e1cb7a988347e68126c7388957"
server_key_iv: "bc1eb3054ba423dfbb4ac94c92ea5e18"
server_key_version: "encryptionkey@0"
I20260621 02:50:14.800884 12893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:14.858942 12893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:14.863292 12893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:14.863608 12893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:14.866600 12893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:14.866739 12893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:14.866950 12893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:14.867122 12893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 21 02:50:14 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:14.891506 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:14.866794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:51705 (local address 127.11.109.62:34177)
0621 02:50:14.867336 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:14.867359 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:14.868709 (+ 1350us) server_negotiation.cc:407] Connection header received
0621 02:50:14.872067 (+ 3358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:14.872074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:14.872164 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:14.872286 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:14.875403 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.876279 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.879913 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.880318 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.880431 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:14.885543 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:14.885548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:14.885558 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:14.885591 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:14.888207 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:14.888896 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:14.888901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:14.888909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:14.889018 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:14.889480 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:14.889485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:14.889489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:14.889855 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:14.890005 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:14.890939 (+ 934us) server_negotiation.cc:299] Negotiation successful
0621 02:50:14.891152 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":236,"threads_started":1}
I20260621 02:50:14.892529 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:14.867483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.11.109.1:51705)
0621 02:50:14.868482 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:14.868550 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:14.871703 (+ 3153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:14.872922 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:14.872939 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:14.874028 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:14.875166 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:14.875206 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.876514 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.876523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:14.879738 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:14.879758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.880609 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.880623 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:14.881017 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:14.881985 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:14.882008 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:14.885315 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:14.888334 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:14.888344 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:14.888359 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:14.888766 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:14.889127 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:14.889131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:14.889135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:14.889363 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:14.890032 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:14.890040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:14.890479 (+ 439us) client_negotiation.cc:769] Sending connection context
0621 02:50:14.890838 (+ 359us) client_negotiation.cc:240] Negotiation successful
0621 02:50:14.891372 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":359,"threads_started":1}
I20260621 02:50:14.916842 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:14.900026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:33045 (local address 127.11.109.62:34177)
0621 02:50:14.900273 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:14.900279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:14.901090 (+ 811us) server_negotiation.cc:407] Connection header received
0621 02:50:14.901374 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:14.901379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:14.901448 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:14.901548 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:14.902441 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.903430 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.905219 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.906072 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.906234 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:14.909782 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:14.909790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:14.909796 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:14.909844 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:14.913765 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:14.914676 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:14.914687 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:14.914693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:14.914865 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:14.915347 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:14.915353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:14.915356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:14.915705 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:14.915861 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:14.916447 (+ 586us) server_negotiation.cc:299] Negotiation successful
0621 02:50:14.916611 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:50:14.917614 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:14.900073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.11.109.1:33045)
0621 02:50:14.900967 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:14.900990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:14.901160 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:14.901576 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:14.901581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:14.902028 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:14.902303 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:14.902312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.903570 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.903575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:14.905029 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:14.905043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:14.906585 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:14.906600 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:14.906884 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:14.907781 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:14.907818 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:14.909501 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:14.913915 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:14.913922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:14.913930 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:14.914493 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:14.915006 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:14.915013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:14.915019 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:14.915205 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:14.915855 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:14.915863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:14.916144 (+ 281us) client_negotiation.cc:769] Sending connection context
0621 02:50:14.916390 (+ 246us) client_negotiation.cc:240] Negotiation successful
0621 02:50:14.916849 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":595,"threads_started":1}
I20260621 02:50:14.925554 12893 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:39289
I20260621 02:50:14.925624 13028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:39289 every 8 connection(s)
I20260621 02:50:14.926785 12893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:14.928781 13029 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34177
I20260621 02:50:14.928988 13029 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:14.929643 13029 heartbeater.cc:511] Master 127.11.109.62:34177 requested a full tablet report, sending...
I20260621 02:50:14.931533 12822 ts_manager.cc:194] Registered new tserver with Master: b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289)
I20260621 02:50:14.933004 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:51705
I20260621 02:50:14.935102 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 12893
I20260621 02:50:14.935302 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:14.935890 11700 external_mini_cluster.cc:1428] Setting key 835b0ccbe150b2a96dccab0ced12a37d
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:14.973037 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34177
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:15.288781 13033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:15.289086 13033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:15.289232 13033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:15.300415 13033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:15.300537 13033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:15.300664 13033 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:50:15.300776 13033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:15.311342 13033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:34177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:15.312573 13033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:15.313910 13033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:15.326300 13038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:15.326375 13041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:15.326316 13039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:15.327401 13033 server_base.cc:1034] running on GCE node
I20260621 02:50:15.327919 13033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:15.328686 13033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:15.329890 13033 hybrid_clock.cc:648] HybridClock initialized: now 1782010215329830 us; error 42 us; skew 500 ppm
I20260621 02:50:15.330137 13033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:15 dist-test-slave-bc52 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:15.336203 13033 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:15.337675 13033 webserver.cc:460] Webserver started at http://127.11.109.2:41041/ using document root <none> and password file <none>
I20260621 02:50:15.338333 13033 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:15.338471 13033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:15.338790 13033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:15.341593 13033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "8c73cbb7a58a4315a9bbd572a2d69136"
format_stamp: "Formatted at 2026-06-21 02:50:15 on dist-test-slave-bc52"
server_key: "e48514aa07829e6673038de652baa1e4"
server_key_iv: "7b97c31daa2a174a108b66625108beae"
server_key_version: "encryptionkey@0"
I20260621 02:50:15.342353 13033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "8c73cbb7a58a4315a9bbd572a2d69136"
format_stamp: "Formatted at 2026-06-21 02:50:15 on dist-test-slave-bc52"
server_key: "e48514aa07829e6673038de652baa1e4"
server_key_iv: "7b97c31daa2a174a108b66625108beae"
server_key_version: "encryptionkey@0"
I20260621 02:50:15.347124 13033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:50:15.350363 13048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:15.351558 13033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260621 02:50:15.351723 13033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "8c73cbb7a58a4315a9bbd572a2d69136"
format_stamp: "Formatted at 2026-06-21 02:50:15 on dist-test-slave-bc52"
server_key: "e48514aa07829e6673038de652baa1e4"
server_key_iv: "7b97c31daa2a174a108b66625108beae"
server_key_version: "encryptionkey@0"
I20260621 02:50:15.351925 13033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:15.372608 13033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:15.376405 13033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:15.376703 13033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:15.379223 13033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:15.379348 13033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:15.379495 13033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:15.379575 13033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:50:15 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:15.402191 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.379919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:60081 (local address 127.11.109.62:34177)
0621 02:50:15.380090 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:15.380094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:15.382009 (+ 1915us) server_negotiation.cc:407] Connection header received
0621 02:50:15.383953 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:15.383962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:15.384045 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:15.384162 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:15.385976 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.387053 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.389701 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.390386 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.390516 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.395199 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:15.395205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:15.395210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:15.395248 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:15.398773 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.399744 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.399753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.399759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.399940 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.400448 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.400453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.400456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.400720 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:15.400886 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:15.401596 (+ 710us) server_negotiation.cc:299] Negotiation successful
0621 02:50:15.401813 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:50:15.403128 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.380774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.11.109.2:60081)
0621 02:50:15.381850 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:15.381884 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:15.383671 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:15.384382 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:15.384403 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:15.384972 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:15.385799 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.385828 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.387196 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.387202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.389577 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.389595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.390641 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.390649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.390928 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.391717 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:15.391757 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:15.394921 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:15.398932 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.398944 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.398977 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.399618 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.400055 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.400061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.400065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.400303 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.400846 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:15.400855 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:15.401233 (+ 378us) client_negotiation.cc:769] Sending connection context
0621 02:50:15.401504 (+ 271us) client_negotiation.cc:240] Negotiation successful
0621 02:50:15.401984 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":520,"threads_started":1}
I20260621 02:50:15.428978 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.416237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.11.109.2:44783)
0621 02:50:15.416901 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:15.416925 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:15.417079 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:15.417465 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:15.417469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:15.417938 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:15.418212 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.418220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.419178 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.419186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.420338 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.420360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.421295 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.421306 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.421455 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.422045 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:15.422061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:15.422740 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:15.426293 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.426299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.426303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.426762 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.427189 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.427193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.427197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.427334 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.427854 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:15.427859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:15.428095 (+ 236us) client_negotiation.cc:769] Sending connection context
0621 02:50:15.428266 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 02:50:15.428586 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":368,"threads_started":1}
I20260621 02:50:15.429307 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.416215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:44783 (local address 127.11.109.62:34177)
0621 02:50:15.416396 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:15.416402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:15.417016 (+ 614us) server_negotiation.cc:407] Connection header received
0621 02:50:15.417271 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:15.417275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:15.417328 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:15.417443 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:15.418346 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.419042 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.420486 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.421083 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.421223 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.422918 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:15.422924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:15.422929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:15.422966 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:15.426171 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.426885 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.426891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.426895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.427052 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.427419 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.427426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.427429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.427741 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:15.427885 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:15.428436 (+ 551us) server_negotiation.cc:299] Negotiation successful
0621 02:50:15.428789 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:15.439697 13033 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:43183
I20260621 02:50:15.439724 13167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:43183 every 8 connection(s)
I20260621 02:50:15.440879 13033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:15.442884 13168 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34177
I20260621 02:50:15.443075 13168 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:15.443614 13168 heartbeater.cc:511] Master 127.11.109.62:34177 requested a full tablet report, sending...
I20260621 02:50:15.444716 12822 ts_manager.cc:194] Registered new tserver with Master: 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183)
I20260621 02:50:15.445616 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:60081
I20260621 02:50:15.449138 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 13033
I20260621 02:50:15.449398 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:15.449827 11700 external_mini_cluster.cc:1428] Setting key ceaf3e802da8b44c5929a7cc78908bce
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:15.486546 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34177
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:15.794473 13172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:15.794831 13172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:15.794979 13172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:15.802134 13172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:15.802266 13172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:15.802397 13172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:50:15.802521 13172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:15.811937 13172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:34177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:15.813092 13172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:15.814375 13172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:15.826725 13177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:15.826758 13180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:15.826673 13178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:15.827435 13172 server_base.cc:1034] running on GCE node
I20260621 02:50:15.827993 13172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:15.828847 13172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:15.830081 13172 hybrid_clock.cc:648] HybridClock initialized: now 1782010215830018 us; error 41 us; skew 500 ppm
I20260621 02:50:15.830379 13172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:15 dist-test-slave-bc52 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:15.833516 13172 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:15.834843 13172 webserver.cc:460] Webserver started at http://127.11.109.3:35499/ using document root <none> and password file <none>
I20260621 02:50:15.835503 13172 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:15.835633 13172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:15.835939 13172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:15.838727 13172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "67d11ec9d5df49b59694a9ca75412933"
format_stamp: "Formatted at 2026-06-21 02:50:15 on dist-test-slave-bc52"
server_key: "a7e189c22b5b98311ece8e0197d4dd13"
server_key_iv: "69da76aa2bb399b7c110eeabbb7b155a"
server_key_version: "encryptionkey@0"
I20260621 02:50:15.839485 13172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "67d11ec9d5df49b59694a9ca75412933"
format_stamp: "Formatted at 2026-06-21 02:50:15 on dist-test-slave-bc52"
server_key: "a7e189c22b5b98311ece8e0197d4dd13"
server_key_iv: "69da76aa2bb399b7c110eeabbb7b155a"
server_key_version: "encryptionkey@0"
I20260621 02:50:15.844273 13172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:50:15.847440 13187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:15.848660 13172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:15.848820 13172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "67d11ec9d5df49b59694a9ca75412933"
format_stamp: "Formatted at 2026-06-21 02:50:15 on dist-test-slave-bc52"
server_key: "a7e189c22b5b98311ece8e0197d4dd13"
server_key_iv: "69da76aa2bb399b7c110eeabbb7b155a"
server_key_version: "encryptionkey@0"
I20260621 02:50:15.849020 13172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:15.882083 13172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:15.886216 13172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:15.886474 13172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:15.889133 13172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:15.889269 13172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 02:50:15.889681 13172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:15.889766 13172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:50:15 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:15.912971 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.889532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:58561 (local address 127.11.109.62:34177)
0621 02:50:15.889703 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:15.889707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:15.891120 (+ 1413us) server_negotiation.cc:407] Connection header received
0621 02:50:15.892984 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:15.892990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:15.893075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:15.893181 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:15.895495 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.896408 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.899695 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.900305 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.900451 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.905790 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:15.905800 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:15.905805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:15.905848 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:15.908652 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.909752 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.909761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.909767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.909948 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.910531 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.910537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.910542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.910962 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:15.911150 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:15.912141 (+ 991us) server_negotiation.cc:299] Negotiation successful
0621 02:50:15.912436 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:50:15.913597 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.890160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.11.109.3:58561)
0621 02:50:15.890901 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:15.890935 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:15.892718 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:15.893419 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:15.893427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:15.894201 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:15.895258 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.895295 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.896604 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.896612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.899531 (+ 2919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.899549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.900591 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.900601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.900932 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.901795 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:15.901826 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:15.905515 (+ 3689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:15.908791 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.908803 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.908824 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.909585 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.910095 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.910101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.910106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.910389 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.911118 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:15.911128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:15.911615 (+ 487us) client_negotiation.cc:769] Sending connection context
0621 02:50:15.911968 (+ 353us) client_negotiation.cc:240] Negotiation successful
0621 02:50:15.912480 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":322,"threads_started":1}
I20260621 02:50:15.934798 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.920305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:45323 (local address 127.11.109.62:34177)
0621 02:50:15.920507 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:15.920516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:15.921092 (+ 576us) server_negotiation.cc:407] Connection header received
0621 02:50:15.921315 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:15.921321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:15.921394 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:15.921476 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:15.922910 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.923583 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.926164 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.926656 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.926801 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.929284 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:15.929291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:15.929295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:15.929337 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:15.932361 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.932962 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.932969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.932972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.933091 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.933441 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.933448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.933452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.933711 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:15.933856 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:15.934227 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 02:50:15.934440 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:50:15.935041 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.920338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.11.109.3:45323)
0621 02:50:15.920984 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:15.921008 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:15.921191 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:15.921711 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:15.921717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:15.922262 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:15.922768 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.922779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.923730 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.923737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.926001 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:15.926048 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.926839 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.926845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:15.927015 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.927694 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:15.927721 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:15.928991 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:15.932482 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.932489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.932493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.932850 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.933213 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:15.933218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:15.933222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:15.933340 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:15.933823 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:15.933828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:15.933996 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 02:50:15.934130 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 02:50:15.934536 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":336,"threads_started":1}
I20260621 02:50:15.935772 13029 heartbeater.cc:503] Master 127.11.109.62:34177 was elected leader, sending a full tablet report...
I20260621 02:50:15.941682 13172 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:40913
I20260621 02:50:15.941704 13306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:40913 every 8 connection(s)
I20260621 02:50:15.942791 13172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:15.944495 13307 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34177
I20260621 02:50:15.944716 13307 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:15.945345 13307 heartbeater.cc:511] Master 127.11.109.62:34177 requested a full tablet report, sending...
I20260621 02:50:15.946524 12822 ts_manager.cc:194] Registered new tserver with Master: 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3:40913)
I20260621 02:50:15.947278 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:58561
I20260621 02:50:15.952234 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 13172
I20260621 02:50:15.952379 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:15.952760 11700 external_mini_cluster.cc:1428] Setting key 8dcba3e80171b21b34e4a42bbdfef739
I20260621 02:50:15.955338 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:50:15 dist-test-slave-bc52 krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:50:15 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:15.998420 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:15.985374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.11.109.62:34177)
0621 02:50:15.985556 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:15.985560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:15.985915 (+ 355us) server_negotiation.cc:407] Connection header received
0621 02:50:15.986056 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:15.986060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:15.986122 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:15.986214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:15.987524 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.988305 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.989585 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:15.989975 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:15.990063 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:15.993620 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:15.993624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:15.993627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:15.993657 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:15.996327 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.996881 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.996885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.996887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.996990 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:15.997338 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:15.997341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:15.997343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:15.997599 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:15.997772 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:15.997980 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 02:50:15.998149 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:16.002079 12822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:16.009402 12822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:16.029510 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.021315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.11.109.2:43183)
0621 02:50:16.021945 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.021953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.021981 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:50:16.022073 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.022080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.022331 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.022524 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.023421 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.024836 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.027560 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.028397 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.028571 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.028775 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.029019 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.029205 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":1035,"threads_started":1}
I20260621 02:50:16.029548 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.020730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:43183 (local address 127.0.0.1:53758)
0621 02:50:16.021479 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.021526 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.021710 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.022804 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.022813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.022850 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.023258 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.023267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.024955 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.024962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.027364 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.027375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.028605 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.028611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.028792 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.028849 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.028981 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.029164 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":937,"threads_started":1}
I20260621 02:50:16.033667 13098 tablet_service.cc:1467] Processing CreateTablet for tablet 26f806b84bf24754a97a500cd49a4992 (DEFAULT_TABLE table=test-table [id=4111875410a24ba782198d83ec5e2ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:16.034929 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.021383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.11.109.1:39289)
0621 02:50:16.023769 (+ 2386us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.023778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.023804 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:50:16.023891 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.023897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.024101 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.024276 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.025101 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.026715 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.030962 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.032936 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.033114 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.033797 (+ 683us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.034485 (+ 688us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.034642 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2241,"thread_start_us":2147,"threads_started":1}
I20260621 02:50:16.035014 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.020806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39289 (local address 127.0.0.1:43692)
0621 02:50:16.022954 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.022977 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.023094 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.024296 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.024304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.024340 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.024863 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.024877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.026883 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.026891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.030736 (+ 3845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.030755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.033994 (+ 3239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.034001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.034217 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.034257 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.034345 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.034602 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2000,"mutex_wait_us":105,"thread_start_us":1847,"threads_started":1}
I20260621 02:50:16.035509 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f806b84bf24754a97a500cd49a4992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.038539 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.022981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:40913 (local address 127.0.0.1:34158)
0621 02:50:16.030263 (+ 7282us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.030279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.030437 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.031356 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.031362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.031388 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.031541 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.031551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.033519 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.033524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.035981 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.036004 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.037752 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.037758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.037940 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.037972 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.038083 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.038228 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7193,"mutex_wait_us":22,"thread_start_us":10627,"threads_started":1}
I20260621 02:50:16.038471 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 26f806b84bf24754a97a500cd49a4992 (DEFAULT_TABLE table=test-table [id=4111875410a24ba782198d83ec5e2ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:16.039616 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.023333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.11.109.3:40913)
0621 02:50:16.023959 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.023970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.030356 (+ 6386us) server_negotiation.cc:407] Connection header received
0621 02:50:16.030632 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.030640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.030855 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.031019 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.031742 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.033323 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.036166 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.037592 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.037719 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.037922 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.038136 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.039272 (+ 1136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":385,"threads_started":1}
I20260621 02:50:16.039855 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f806b84bf24754a97a500cd49a4992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.043036 13237 tablet_service.cc:1467] Processing CreateTablet for tablet 26f806b84bf24754a97a500cd49a4992 (DEFAULT_TABLE table=test-table [id=4111875410a24ba782198d83ec5e2ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:16.044998 13237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f806b84bf24754a97a500cd49a4992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.046911 13326 tablet_bootstrap.cc:492] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Bootstrap starting.
I20260621 02:50:16.049916 13326 tablet_bootstrap.cc:654] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.050882 13326 log.cc:824] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:16.054941 13326 tablet_bootstrap.cc:492] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: No bootstrap required, opened a new log
I20260621 02:50:16.055239 13326 ts_tablet_manager.cc:1397] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:50:16.057639 13329 tablet_bootstrap.cc:492] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Bootstrap starting.
I20260621 02:50:16.060314 13329 tablet_bootstrap.cc:654] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.060629 13328 tablet_bootstrap.cc:492] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Bootstrap starting.
I20260621 02:50:16.061618 13329 log.cc:824] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:16.063349 13326 raft_consensus.cc:348] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.063696 13326 raft_consensus.cc:374] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.063819 13326 raft_consensus.cc:729] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c73cbb7a58a4315a9bbd572a2d69136, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.064059 13328 tablet_bootstrap.cc:654] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.064414 13326 consensus_queue.cc:260] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.064613 13329 tablet_bootstrap.cc:492] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: No bootstrap required, opened a new log
I20260621 02:50:16.064860 13329 ts_tablet_manager.cc:1397] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260621 02:50:16.065498 13168 heartbeater.cc:503] Master 127.11.109.62:34177 was elected leader, sending a full tablet report...
I20260621 02:50:16.065996 13326 ts_tablet_manager.cc:1428] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260621 02:50:16.065263 13328 log.cc:824] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:16.068511 13328 tablet_bootstrap.cc:492] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: No bootstrap required, opened a new log
I20260621 02:50:16.068765 13328 ts_tablet_manager.cc:1397] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20260621 02:50:16.073804 13329 raft_consensus.cc:348] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.074124 13329 raft_consensus.cc:374] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.074239 13329 raft_consensus.cc:729] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d11ec9d5df49b59694a9ca75412933, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.074807 13329 consensus_queue.cc:260] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.075806 13307 heartbeater.cc:503] Master 127.11.109.62:34177 was elected leader, sending a full tablet report...
I20260621 02:50:16.076331 13329 ts_tablet_manager.cc:1428] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260621 02:50:16.076844 13328 raft_consensus.cc:348] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.077234 13328 raft_consensus.cc:374] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.077350 13328 raft_consensus.cc:729] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cff584bd8f43b896dc47eebde0d997, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.077898 13328 consensus_queue.cc:260] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.079413 13328 ts_tablet_manager.cc:1428] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260621 02:50:16.145159 13333 raft_consensus.cc:482] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:16.145503 13333 raft_consensus.cc:504] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.147149 13333 leader_election.cc:288] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183), b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289)
I20260621 02:50:16.153990 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.147728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:58325 (local address 127.11.109.2:43183)
0621 02:50:16.147978 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.147987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.148016 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:50:16.148183 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.148190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.148273 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.148371 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.148825 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.149881 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.152155 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.153079 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.153263 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.153339 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.153626 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.153746 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 02:50:16.153999 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.147591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:43183 (local address 127.11.109.3:58325)
0621 02:50:16.147848 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.147872 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.147999 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.148413 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.148420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.148454 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.148678 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.148691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.150013 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.150016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.152001 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.152017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.153235 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.153243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.153425 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.153449 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.153577 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.153712 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":3}
I20260621 02:50:16.154213 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.147577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:53409 (local address 127.11.109.1:39289)
0621 02:50:16.147948 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.147954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.148553 (+ 599us) server_negotiation.cc:407] Connection header received
0621 02:50:16.148739 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.148742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.148836 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.148933 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.149423 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.150272 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.152504 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.153313 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.153485 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.153560 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.153846 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.153969 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260621 02:50:16.154361 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.147650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39289 (local address 127.11.109.3:53409)
0621 02:50:16.148446 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.148462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.148614 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.148949 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.148951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.148970 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.149289 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.149298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.150387 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.150392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.152346 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.152356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.153528 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.153543 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.153688 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.153710 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.153818 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.153953 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":50,"thread_start_us":469,"threads_started":1}
I20260621 02:50:16.155134 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f806b84bf24754a97a500cd49a4992" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" is_pre_election: true
I20260621 02:50:16.155297 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f806b84bf24754a97a500cd49a4992" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cff584bd8f43b896dc47eebde0d997" is_pre_election: true
I20260621 02:50:16.155774 12984 raft_consensus.cc:2454] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 0.
I20260621 02:50:16.155774 13122 raft_consensus.cc:2454] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 0.
I20260621 02:50:16.156425 13191 leader_election.cc:302] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d11ec9d5df49b59694a9ca75412933, b7cff584bd8f43b896dc47eebde0d997; no voters:
I20260621 02:50:16.156801 13333 raft_consensus.cc:2781] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:16.156932 13333 raft_consensus.cc:482] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:16.157051 13333 raft_consensus.cc:3037] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.159209 13333 raft_consensus.cc:504] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.159827 13333 leader_election.cc:288] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 election: Requested vote from peers 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183), b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289)
I20260621 02:50:16.160149 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f806b84bf24754a97a500cd49a4992" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c73cbb7a58a4315a9bbd572a2d69136"
I20260621 02:50:16.160339 13122 raft_consensus.cc:3037] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.160312 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f806b84bf24754a97a500cd49a4992" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cff584bd8f43b896dc47eebde0d997"
I20260621 02:50:16.160516 12984 raft_consensus.cc:3037] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.162572 13122 raft_consensus.cc:2454] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 1.
I20260621 02:50:16.162627 12984 raft_consensus.cc:2454] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 1.
I20260621 02:50:16.162962 13190 leader_election.cc:302] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d11ec9d5df49b59694a9ca75412933, 8c73cbb7a58a4315a9bbd572a2d69136; no voters:
I20260621 02:50:16.163261 13333 raft_consensus.cc:2781] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:16.163638 13333 raft_consensus.cc:686] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 1 LEADER]: Becoming Leader. State: Replica: 67d11ec9d5df49b59694a9ca75412933, State: Running, Role: LEADER
I20260621 02:50:16.164076 13333 consensus_queue.cc:237] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.168601 12822 catalog_manager.cc:5495] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } health_report { overall_health: HEALTHY } } }
W20260621 02:50:16.180343 13030 tablet.cc:2369] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:50:16.190187 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.184738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.11.109.3:40913)
0621 02:50:16.184936 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.184944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.185096 (+ 152us) server_negotiation.cc:407] Connection header received
0621 02:50:16.185444 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.185448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.185525 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.185667 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:16.186018 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.186704 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.188008 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.188419 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.188541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.189081 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.189591 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.189743 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.189862 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.189938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 02:50:16.192808 13169 tablet.cc:2369] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:50:16.194430 13308 tablet.cc:2369] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:50:16.197700 12984 raft_consensus.cc:1261] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Refusing update from remote peer 67d11ec9d5df49b59694a9ca75412933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:16.197711 13122 raft_consensus.cc:1261] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Refusing update from remote peer 67d11ec9d5df49b59694a9ca75412933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:16.198589 13336 consensus_queue.cc:1035] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:16.198910 13333 consensus_queue.cc:1035] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:16.228345 12821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:16.228590 12821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:16.232299 12821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: Sending DeleteTablet for 3 replicas of tablet 26f806b84bf24754a97a500cd49a4992
I20260621 02:50:16.233376 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f806b84bf24754a97a500cd49a4992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:16 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:53758
I20260621 02:50:16.233690 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f806b84bf24754a97a500cd49a4992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:16 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:43692
I20260621 02:50:16.234054 13237 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f806b84bf24754a97a500cd49a4992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:16 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:34158
I20260621 02:50:16.234557 13350 tablet_replica.cc:330] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: stopping tablet replica
I20260621 02:50:16.234956 13350 raft_consensus.cc:2229] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:16.235424 13350 raft_consensus.cc:2258] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:16.235771 12821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:16.236531 12821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:16.237515 13350 ts_tablet_manager.cc:1905] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:16.239205 13349 tablet_replica.cc:330] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: stopping tablet replica
I20260621 02:50:16.239665 13349 raft_consensus.cc:2229] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:16.240203 13349 raft_consensus.cc:2258] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:16.244320 13350 ts_tablet_manager.cc:1918] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:16.244465 13350 log.cc:1196] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/26f806b84bf24754a97a500cd49a4992
I20260621 02:50:16.244969 13350 ts_tablet_manager.cc:1939] T 26f806b84bf24754a97a500cd49a4992 P b7cff584bd8f43b896dc47eebde0d997: Deleting consensus metadata
I20260621 02:50:16.245906 12959 tablet_service.cc:1467] Processing CreateTablet for tablet dfb96491c8bf466faf617fcb717941cf (DEFAULT_TABLE table=test-table [id=50413eada6c14ff2abe6a1198cb069da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:16.246345 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb96491c8bf466faf617fcb717941cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.246435 13098 tablet_service.cc:1467] Processing CreateTablet for tablet dfb96491c8bf466faf617fcb717941cf (DEFAULT_TABLE table=test-table [id=50413eada6c14ff2abe6a1198cb069da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:16.246847 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb96491c8bf466faf617fcb717941cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.246905 12809 catalog_manager.cc:4841] TS b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289): tablet 26f806b84bf24754a97a500cd49a4992 (table test-table [id=4111875410a24ba782198d83ec5e2ee4]) successfully deleted
I20260621 02:50:16.247675 13236 tablet_service.cc:1467] Processing CreateTablet for tablet dfb96491c8bf466faf617fcb717941cf (DEFAULT_TABLE table=test-table [id=50413eada6c14ff2abe6a1198cb069da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:16.248199 13236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb96491c8bf466faf617fcb717941cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.249585 13349 ts_tablet_manager.cc:1905] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:16.252429 13328 tablet_bootstrap.cc:492] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Bootstrap starting.
I20260621 02:50:16.253543 13326 tablet_bootstrap.cc:492] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Bootstrap starting.
I20260621 02:50:16.254061 13328 tablet_bootstrap.cc:654] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.254699 13329 tablet_bootstrap.cc:492] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Bootstrap starting.
I20260621 02:50:16.255223 13326 tablet_bootstrap.cc:654] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.256596 13329 tablet_bootstrap.cc:654] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.266079 13328 tablet_bootstrap.cc:492] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: No bootstrap required, opened a new log
I20260621 02:50:16.266234 13328 ts_tablet_manager.cc:1397] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260621 02:50:16.267016 13328 raft_consensus.cc:348] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.267256 13328 raft_consensus.cc:374] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.267344 13328 raft_consensus.cc:729] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cff584bd8f43b896dc47eebde0d997, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.267642 13328 consensus_queue.cc:260] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.269814 13326 tablet_bootstrap.cc:492] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: No bootstrap required, opened a new log
I20260621 02:50:16.269829 13329 tablet_bootstrap.cc:492] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: No bootstrap required, opened a new log
I20260621 02:50:16.270071 13326 ts_tablet_manager.cc:1397] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20260621 02:50:16.270195 13329 ts_tablet_manager.cc:1397] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20260621 02:50:16.270613 13328 ts_tablet_manager.cc:1428] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260621 02:50:16.270920 13326 raft_consensus.cc:348] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.271135 13326 raft_consensus.cc:374] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.271240 13326 raft_consensus.cc:729] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c73cbb7a58a4315a9bbd572a2d69136, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.271258 13329 raft_consensus.cc:348] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.271593 13329 raft_consensus.cc:374] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.271528 13326 consensus_queue.cc:260] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.271731 13329 raft_consensus.cc:729] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d11ec9d5df49b59694a9ca75412933, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.272114 13329 consensus_queue.cc:260] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.272931 13326 ts_tablet_manager.cc:1428] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:16.273589 13351 tablet_replica.cc:330] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: stopping tablet replica
I20260621 02:50:16.274001 13351 raft_consensus.cc:2229] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:16.274605 13351 raft_consensus.cc:2258] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:16.275450 13329 ts_tablet_manager.cc:1428] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260621 02:50:16.276492 13349 ts_tablet_manager.cc:1918] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:16.276736 13351 ts_tablet_manager.cc:1905] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:16.276752 13349 log.cc:1196] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/26f806b84bf24754a97a500cd49a4992
I20260621 02:50:16.277565 13349 ts_tablet_manager.cc:1939] T 26f806b84bf24754a97a500cd49a4992 P 8c73cbb7a58a4315a9bbd572a2d69136: Deleting consensus metadata
I20260621 02:50:16.279145 12808 catalog_manager.cc:4841] TS 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183): tablet 26f806b84bf24754a97a500cd49a4992 (table test-table [id=4111875410a24ba782198d83ec5e2ee4]) successfully deleted
I20260621 02:50:16.283957 13351 ts_tablet_manager.cc:1918] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:16.284107 13351 log.cc:1196] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/26f806b84bf24754a97a500cd49a4992
I20260621 02:50:16.284626 13351 ts_tablet_manager.cc:1939] T 26f806b84bf24754a97a500cd49a4992 P 67d11ec9d5df49b59694a9ca75412933: Deleting consensus metadata
I20260621 02:50:16.286206 12809 catalog_manager.cc:4841] TS 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3:40913): tablet 26f806b84bf24754a97a500cd49a4992 (table test-table [id=4111875410a24ba782198d83ec5e2ee4]) successfully deleted
I20260621 02:50:16.397330 13334 raft_consensus.cc:482] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:16.397579 13334 raft_consensus.cc:504] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.399070 13334 leader_election.cc:288] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183), 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3:40913)
I20260621 02:50:16.407146 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.399931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:42021 (local address 127.11.109.2:43183)
0621 02:50:16.400096 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.400102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.400391 (+ 289us) server_negotiation.cc:407] Connection header received
0621 02:50:16.400714 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.400721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.400791 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.400888 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.401527 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.402671 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.405194 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.406130 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.406284 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.406333 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.406731 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.406849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:50:16.407275 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.399453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:43183 (local address 127.11.109.1:42021)
0621 02:50:16.400248 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.400273 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.400513 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.400911 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.400916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.400940 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.401349 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.401363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.402802 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.402809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.404997 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.405014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.406445 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.406449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.406567 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.406599 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.406708 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.406836 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":4,"thread_start_us":770,"threads_started":1}
I20260621 02:50:16.407814 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.399485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:51599 (local address 127.11.109.3:40913)
0621 02:50:16.399674 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.399683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.400619 (+ 936us) server_negotiation.cc:407] Connection header received
0621 02:50:16.400765 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.400771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.400853 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.400963 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:16.401700 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.402428 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.405257 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.406605 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.406747 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.406815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.407382 (+ 567us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.407526 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:50:16.408051 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb96491c8bf466faf617fcb717941cf" candidate_uuid: "b7cff584bd8f43b896dc47eebde0d997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" is_pre_election: true
I20260621 02:50:16.408246 13122 raft_consensus.cc:2454] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7cff584bd8f43b896dc47eebde0d997 in term 0.
I20260621 02:50:16.408494 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.399493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:40913 (local address 127.11.109.1:51599)
0621 02:50:16.400512 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.400527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.400641 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.401096 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.401100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.401119 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:16.401532 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.401542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.402556 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.402562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.405010 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.405018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.406996 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.407004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.407186 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.407224 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.407990 (+ 766us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.408120 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":122,"thread_start_us":417,"threads_started":1}
I20260621 02:50:16.408756 12911 leader_election.cc:302] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c73cbb7a58a4315a9bbd572a2d69136, b7cff584bd8f43b896dc47eebde0d997; no voters:
I20260621 02:50:16.409194 13334 raft_consensus.cc:2781] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:16.409319 13334 raft_consensus.cc:482] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:16.409432 13334 raft_consensus.cc:3037] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.409513 13258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb96491c8bf466faf617fcb717941cf" candidate_uuid: "b7cff584bd8f43b896dc47eebde0d997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d11ec9d5df49b59694a9ca75412933" is_pre_election: true
I20260621 02:50:16.409929 13258 raft_consensus.cc:2454] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7cff584bd8f43b896dc47eebde0d997 in term 0.
I20260621 02:50:16.411707 13334 raft_consensus.cc:504] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.412276 13334 leader_election.cc:288] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [CANDIDATE]: Term 1 election: Requested vote from peers 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183), 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3:40913)
I20260621 02:50:16.412659 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb96491c8bf466faf617fcb717941cf" candidate_uuid: "b7cff584bd8f43b896dc47eebde0d997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c73cbb7a58a4315a9bbd572a2d69136"
I20260621 02:50:16.412782 13258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb96491c8bf466faf617fcb717941cf" candidate_uuid: "b7cff584bd8f43b896dc47eebde0d997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d11ec9d5df49b59694a9ca75412933"
I20260621 02:50:16.412842 13122 raft_consensus.cc:3037] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.412979 13258 raft_consensus.cc:3037] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.414803 13122 raft_consensus.cc:2454] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7cff584bd8f43b896dc47eebde0d997 in term 1.
I20260621 02:50:16.415162 12911 leader_election.cc:302] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c73cbb7a58a4315a9bbd572a2d69136, b7cff584bd8f43b896dc47eebde0d997; no voters:
I20260621 02:50:16.415472 13334 raft_consensus.cc:2781] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:16.415952 13334 raft_consensus.cc:686] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 1 LEADER]: Becoming Leader. State: Replica: b7cff584bd8f43b896dc47eebde0d997, State: Running, Role: LEADER
I20260621 02:50:16.416276 13258 raft_consensus.cc:2454] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7cff584bd8f43b896dc47eebde0d997 in term 1.
I20260621 02:50:16.416386 13334 consensus_queue.cc:237] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.419850 12821 catalog_manager.cc:5495] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1). New cstate: current_term: 1 leader_uuid: "b7cff584bd8f43b896dc47eebde0d997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } health_report { overall_health: UNKNOWN } } }
Jun 21 02:50:16 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010214, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:16.490100 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.475687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.11.109.62:34177)
0621 02:50:16.475854 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.475859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.476361 (+ 502us) server_negotiation.cc:407] Connection header received
0621 02:50:16.476548 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.476552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.476634 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.476736 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:16.477885 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.478673 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.479926 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.480336 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.480485 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.483846 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:16.483855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:16.483861 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:16.483899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:16.487208 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.487888 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.487894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.487898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.488060 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.488419 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.488425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.488428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.488789 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:16.488942 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.489388 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.489694 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":54}
I20260621 02:50:16.490134 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.475675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34177 (local address 127.0.0.1:42246)
0621 02:50:16.476250 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.476265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.476428 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.476762 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.476767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.477477 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:16.477731 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.477740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.478797 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.478803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.479788 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.479805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.480525 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.480531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.480705 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.481407 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:16.481424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:16.483658 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:16.487379 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:16.487388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:16.487394 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:16.487771 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:16.488169 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:16.488172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:16.488175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:16.488303 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:16.488915 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:16.488920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:16.489061 (+ 141us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.489289 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.489572 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":335,"threads_started":1}
I20260621 02:50:16.495204 12821 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:42246:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:50:16.496361 12821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:16.502933 13098 tablet_service.cc:1467] Processing CreateTablet for tablet bae2217041bd49369f2224c2a3543082 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90ac1fe0f1ee465b860e50a81fa42b7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:50:16.502944 12959 tablet_service.cc:1467] Processing CreateTablet for tablet bae2217041bd49369f2224c2a3543082 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90ac1fe0f1ee465b860e50a81fa42b7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:50:16.503438 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae2217041bd49369f2224c2a3543082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.503479 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae2217041bd49369f2224c2a3543082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.508736 13326 tablet_bootstrap.cc:492] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: Bootstrap starting.
I20260621 02:50:16.510085 13237 tablet_service.cc:1467] Processing CreateTablet for tablet bae2217041bd49369f2224c2a3543082 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90ac1fe0f1ee465b860e50a81fa42b7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:50:16.510622 13326 tablet_bootstrap.cc:654] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.510658 13237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae2217041bd49369f2224c2a3543082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:16.512760 13326 tablet_bootstrap.cc:492] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: No bootstrap required, opened a new log
I20260621 02:50:16.512921 13326 ts_tablet_manager.cc:1397] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:50:16.513680 13326 raft_consensus.cc:348] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.513913 13326 raft_consensus.cc:374] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.513995 13326 raft_consensus.cc:729] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c73cbb7a58a4315a9bbd572a2d69136, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.514266 13326 consensus_queue.cc:260] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.514863 13328 tablet_bootstrap.cc:492] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: Bootstrap starting.
I20260621 02:50:16.514681 13331 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.514993 13331 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.516592 13328 tablet_bootstrap.cc:654] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.517230 13326 ts_tablet_manager.cc:1428] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:50:16.520970 13329 tablet_bootstrap.cc:492] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: Bootstrap starting.
I20260621 02:50:16.522269 13329 tablet_bootstrap.cc:654] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:16.524871 13329 tablet_bootstrap.cc:492] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: No bootstrap required, opened a new log
I20260621 02:50:16.525046 13329 ts_tablet_manager.cc:1397] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:50:16.525947 13329 raft_consensus.cc:348] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.526276 13329 raft_consensus.cc:374] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.526383 13329 raft_consensus.cc:729] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d11ec9d5df49b59694a9ca75412933, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.526680 13328 tablet_bootstrap.cc:492] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: No bootstrap required, opened a new log
I20260621 02:50:16.526651 13329 consensus_queue.cc:260] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.526854 13328 ts_tablet_manager.cc:1397] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260621 02:50:16.527066 13333 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.527377 13333 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.527592 13328 raft_consensus.cc:348] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.527827 13328 raft_consensus.cc:374] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:16.527908 13328 raft_consensus.cc:729] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cff584bd8f43b896dc47eebde0d997, State: Initialized, Role: FOLLOWER
I20260621 02:50:16.528184 13328 consensus_queue.cc:260] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.528614 13334 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.528877 13334 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.529857 13329 ts_tablet_manager.cc:1428] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260621 02:50:16.530162 13328 ts_tablet_manager.cc:1428] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260621 02:50:16.609604 13333 raft_consensus.cc:482] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:16.609853 13333 raft_consensus.cc:504] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.610539 13333 leader_election.cc:288] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289), 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183)
I20260621 02:50:16.610937 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae2217041bd49369f2224c2a3543082" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cff584bd8f43b896dc47eebde0d997" is_pre_election: true
I20260621 02:50:16.611218 12984 raft_consensus.cc:2454] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 0.
I20260621 02:50:16.611151 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae2217041bd49369f2224c2a3543082" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" is_pre_election: true
I20260621 02:50:16.611402 13122 raft_consensus.cc:2454] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 0.
I20260621 02:50:16.611625 13191 leader_election.cc:302] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d11ec9d5df49b59694a9ca75412933, b7cff584bd8f43b896dc47eebde0d997; no voters:
I20260621 02:50:16.611905 13333 raft_consensus.cc:2781] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:16.612043 13333 raft_consensus.cc:482] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:16.612142 13333 raft_consensus.cc:3037] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.615398 13333 raft_consensus.cc:504] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.616027 13333 leader_election.cc:288] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 election: Requested vote from peers b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289), 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183)
I20260621 02:50:16.616482 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae2217041bd49369f2224c2a3543082" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c73cbb7a58a4315a9bbd572a2d69136"
I20260621 02:50:16.616566 12984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae2217041bd49369f2224c2a3543082" candidate_uuid: "67d11ec9d5df49b59694a9ca75412933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cff584bd8f43b896dc47eebde0d997"
I20260621 02:50:16.616711 13122 raft_consensus.cc:3037] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.616822 12984 raft_consensus.cc:3037] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:16.618848 12984 raft_consensus.cc:2454] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 1.
I20260621 02:50:16.619244 13191 leader_election.cc:302] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d11ec9d5df49b59694a9ca75412933, b7cff584bd8f43b896dc47eebde0d997; no voters:
I20260621 02:50:16.619529 13333 raft_consensus.cc:2781] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:16.619602 13122 raft_consensus.cc:2454] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d11ec9d5df49b59694a9ca75412933 in term 1.
I20260621 02:50:16.619706 13333 raft_consensus.cc:686] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [term 1 LEADER]: Becoming Leader. State: Replica: 67d11ec9d5df49b59694a9ca75412933, State: Running, Role: LEADER
I20260621 02:50:16.620059 13333 consensus_queue.cc:237] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } }
I20260621 02:50:16.621910 13344 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: TxnStatusTablet state changed. Reason: New leader 67d11ec9d5df49b59694a9ca75412933. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.622372 13344 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:50:16.623140 13369 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:50:16.623396 13369 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:50:16.623818 12822 catalog_manager.cc:5495] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } health_report { overall_health: HEALTHY } } }
I20260621 02:50:16.645067 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.639647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.11.109.3:40913)
0621 02:50:16.640181 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.640186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.640200 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:50:16.640267 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.640270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.640326 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.640413 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:16.640790 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.641595 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.643518 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.643921 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.644009 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.644498 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.644589 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.644653 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.644810 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.644866 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420}
I20260621 02:50:16.645258 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.639614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:40913 (local address 127.0.0.1:34174)
0621 02:50:16.639854 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.639871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.639972 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.640462 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.640466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.640481 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:16.640662 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.640671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.641724 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.641730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.643374 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.643391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.644069 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.644075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.644251 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.644320 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.644685 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.644703 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.644785 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.644930 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:50:16.648720 13122 raft_consensus.cc:1261] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Refusing update from remote peer 67d11ec9d5df49b59694a9ca75412933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:16.648753 12984 raft_consensus.cc:1261] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Refusing update from remote peer 67d11ec9d5df49b59694a9ca75412933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:16.649374 13344 consensus_queue.cc:1035] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:16.649698 13333 consensus_queue.cc:1035] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:16.653138 13334 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: TxnStatusTablet state changed. Reason: New leader 67d11ec9d5df49b59694a9ca75412933. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.654310 13334 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.656757 13344 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.656991 13344 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:50:16.657203 13339 mvcc.cc:205] Tried to move back new op lower bound from 7299113847387447296 to 7299113847278358528. Current Snapshot: MvccSnapshot[applied={T|T < 7299113847387447296 or (T in {7299113847387447296})}]
I20260621 02:50:16.658293 13340 mvcc.cc:205] Tried to move back new op lower bound from 7299113847387447296 to 7299113847278358528. Current Snapshot: MvccSnapshot[applied={T|T < 7299113847387447296}]
I20260621 02:50:16.658560 13334 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.658797 13334 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P b7cff584bd8f43b896dc47eebde0d997: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.667932 13333 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.667935 13331 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: TxnStatusTablet state changed. Reason: New leader 67d11ec9d5df49b59694a9ca75412933. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.668201 13331 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.668197 13333 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 67d11ec9d5df49b59694a9ca75412933: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:50:16.670775 13331 tablet_replica.cc:439] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67d11ec9d5df49b59694a9ca75412933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cff584bd8f43b896dc47eebde0d997" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39289 } } peers { permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 } } peers { permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 } } }
I20260621 02:50:16.670972 13331 tablet_replica.cc:442] T bae2217041bd49369f2224c2a3543082 P 8c73cbb7a58a4315a9bbd572a2d69136: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:16.674517 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.664951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.11.109.1:39289)
0621 02:50:16.666333 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.666339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.666357 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:50:16.666445 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.666450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.666554 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.666713 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:16.667184 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.670044 (+ 2860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.672043 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.672662 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.672782 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.673234 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.673902 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.674038 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.674140 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.674220 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1133,"threads_started":1}
I20260621 02:50:16.691045 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.684690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:51997 (local address 127.11.109.3:40913)
0621 02:50:16.684936 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.684946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.685486 (+ 540us) server_negotiation.cc:407] Connection header received
0621 02:50:16.685666 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.685671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.685736 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.685808 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:16.686369 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.687152 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.689234 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.689768 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.689861 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.690427 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.690535 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.690600 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.690751 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.690810 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:50:16.691247 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.684690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:40913 (local address 127.11.109.1:51997)
0621 02:50:16.685383 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.685403 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.685539 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.685853 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.685860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.685874 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:16.686222 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.686233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.687274 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.687280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.689037 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.689052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.689944 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.689953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.690140 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.690209 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.690626 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.690638 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.690729 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.690863 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":375,"threads_started":1}
I20260621 02:50:16.700881 13258 raft_consensus.cc:1261] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Refusing update from remote peer b7cff584bd8f43b896dc47eebde0d997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:16.701818 13121 raft_consensus.cc:1261] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Refusing update from remote peer b7cff584bd8f43b896dc47eebde0d997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:16.702195 13357 consensus_queue.cc:1035] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d11ec9d5df49b59694a9ca75412933" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:16.702769 13334 consensus_queue.cc:1035] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c73cbb7a58a4315a9bbd572a2d69136" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 43183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:16.715008 13341 mvcc.cc:205] Tried to move back new op lower bound from 7299113847599489024 to 7299113846445522944. Current Snapshot: MvccSnapshot[applied={T|T < 7299113847599489024}]
I20260621 02:50:16.741830 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.735885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54499 (local address 127.11.109.1:39289)
0621 02:50:16.736032 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.736036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.736588 (+ 552us) server_negotiation.cc:407] Connection header received
0621 02:50:16.736794 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.736800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.736866 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.736957 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:16.737512 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.738369 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.739935 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.740334 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.740502 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.741017 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.741156 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.741278 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.741433 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.741514 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:50:16.741959 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.735905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39289 (local address 127.11.109.3:54499)
0621 02:50:16.736488 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:16.736504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:16.736653 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:16.736984 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:16.736989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:16.737002 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:16.737361 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.737372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.738497 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.738503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.739800 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:16.739813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.740524 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.740530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:16.740734 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.740822 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:16.741295 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:16.741306 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:50:16.741398 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:50:16.741555 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
I20260621 02:50:16.772866 12822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:16.773130 12822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:16.777465 12822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9090a2620e3e4f69b3e15bac50780a50: Sending DeleteTablet for 3 replicas of tablet dfb96491c8bf466faf617fcb717941cf
I20260621 02:50:16.778434 13237 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb96491c8bf466faf617fcb717941cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:16 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:34158
I20260621 02:50:16.778628 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb96491c8bf466faf617fcb717941cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:16 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:53758
I20260621 02:50:16.778704 13351 tablet_replica.cc:330] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: stopping tablet replica
I20260621 02:50:16.778891 13351 raft_consensus.cc:2229] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:16.779063 13351 raft_consensus.cc:2258] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:16.779438 13349 tablet_replica.cc:330] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: stopping tablet replica
I20260621 02:50:16.779418 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb96491c8bf466faf617fcb717941cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:16 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:43692
I20260621 02:50:16.779623 13349 raft_consensus.cc:2229] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:16.779788 13349 raft_consensus.cc:2258] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:16.781004 13351 ts_tablet_manager.cc:1905] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:16.786269 13349 ts_tablet_manager.cc:1905] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:16.796347 13349 ts_tablet_manager.cc:1918] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:16.796993 13349 log.cc:1196] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/dfb96491c8bf466faf617fcb717941cf
I20260621 02:50:16.797714 13351 ts_tablet_manager.cc:1918] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:16.798293 13349 ts_tablet_manager.cc:1939] T dfb96491c8bf466faf617fcb717941cf P 8c73cbb7a58a4315a9bbd572a2d69136: Deleting consensus metadata
I20260621 02:50:16.799459 12808 catalog_manager.cc:4841] TS 8c73cbb7a58a4315a9bbd572a2d69136 (127.11.109.2:43183): tablet dfb96491c8bf466faf617fcb717941cf (table test-table [id=50413eada6c14ff2abe6a1198cb069da]) successfully deleted
I20260621 02:50:16.799798 13351 log.cc:1196] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/dfb96491c8bf466faf617fcb717941cf
I20260621 02:50:16.800326 13351 ts_tablet_manager.cc:1939] T dfb96491c8bf466faf617fcb717941cf P 67d11ec9d5df49b59694a9ca75412933: Deleting consensus metadata
I20260621 02:50:16.801451 12809 catalog_manager.cc:4841] TS 67d11ec9d5df49b59694a9ca75412933 (127.11.109.3:40913): tablet dfb96491c8bf466faf617fcb717941cf (table test-table [id=50413eada6c14ff2abe6a1198cb069da]) successfully deleted
I20260621 02:50:16.802194 13385 tablet_replica.cc:330] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: stopping tablet replica
I20260621 02:50:16.802742 13385 raft_consensus.cc:2229] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:16.803365 13385 raft_consensus.cc:2258] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:16.804617 13385 ts_tablet_manager.cc:1905] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:16.812103 13385 ts_tablet_manager.cc:1918] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:16.812244 13385 log.cc:1196] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/dfb96491c8bf466faf617fcb717941cf
I20260621 02:50:16.812740 13385 ts_tablet_manager.cc:1939] T dfb96491c8bf466faf617fcb717941cf P b7cff584bd8f43b896dc47eebde0d997: Deleting consensus metadata
I20260621 02:50:16.813894 12809 catalog_manager.cc:4841] TS b7cff584bd8f43b896dc47eebde0d997 (127.11.109.1:39289): tablet dfb96491c8bf466faf617fcb717941cf (table test-table [id=50413eada6c14ff2abe6a1198cb069da]) successfully deleted
Jun 21 02:50:16 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:16.836933 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.804228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.11.109.1:39289)
0621 02:50:16.804408 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.804415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.817840 (+ 13425us) server_negotiation.cc:407] Connection header received
0621 02:50:16.817991 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.817996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.818098 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.818218 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:16.820100 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.821439 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.823406 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.824119 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.824252 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.829524 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:16.829548 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:16.829573 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:16.829634 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:16.833826 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.834651 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.834661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.834665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.834821 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.835218 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.835223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.835226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.835676 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:16.836007 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.836274 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.836497 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260621 02:50:16.837962 13024 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43700
I20260621 02:50:16.858211 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.841456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.11.109.62:34177)
0621 02:50:16.841659 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.841665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.842140 (+ 475us) server_negotiation.cc:407] Connection header received
0621 02:50:16.842388 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.842394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.842474 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.842546 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:16.844367 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.845552 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.847478 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.848165 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.848295 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.851328 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:16.851335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:16.851340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:16.851380 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:16.854912 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.855729 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.855735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.855739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.855977 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.856387 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.856392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.856395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.856769 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:16.857032 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.857620 (+ 588us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.857864 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 02:50:16.859040 12822 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42262
I20260621 02:50:16.862119 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12893
I20260621 02:50:16.878608 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.863185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.11.109.1:39289)
0621 02:50:16.863400 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.863407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.863561 (+ 154us) server_negotiation.cc:407] Connection header received
0621 02:50:16.863789 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.863795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.863877 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.863944 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:16.865584 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.866656 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.868394 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.869034 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.869164 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.871924 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:16.871930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:16.871937 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:16.871984 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:16.875689 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.876467 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.876472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.876476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.876622 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.877022 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.877027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.877031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.877419 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:16.877706 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.877966 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.878228 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 02:50:16.879434 13024 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706
W20260621 02:50:16.880082 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:16.880263 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12893
I20260621 02:50:16.898326 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13033
Jun 21 02:50:16 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:16.913781 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.899139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.11.109.2:43183)
0621 02:50:16.899344 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.899352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.899372 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:50:16.899496 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.899502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.899600 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.899707 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:16.900843 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.901978 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.903185 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.903825 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.903964 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.908384 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:16.908397 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:16.908413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:16.908445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:16.911156 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.911933 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.911940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.911942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.912072 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.912449 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.912452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.912453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.912755 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:16.912969 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.913347 (+ 378us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.913515 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 02:50:16.914593 13163 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53762
W20260621 02:50:16.915210 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13033: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:16.915378 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 13033
I20260621 02:50:16.936183 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13172
Jun 21 02:50:16 dist-test-slave-bc52 krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:16.955157 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:16.936970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.11.109.3:40913)
0621 02:50:16.937254 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:16.937259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:16.937276 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:50:16.937409 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:16.937414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:16.937479 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:16.937561 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:16.939083 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.939792 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.941619 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:16.942293 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:16.942415 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:16.947812 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:16.947826 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:16.947844 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:16.947892 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:16.951969 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.952790 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.952799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.952803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.952952 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:16.953436 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:16.953444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:16.953452 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:16.953912 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:16.954263 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:16.954543 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 02:50:16.954785 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260621 02:50:16.955765 13302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34182
W20260621 02:50:16.956521 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:16.956666 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 13172
I20260621 02:50:16.974771 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 12790
I20260621 02:50:16.975755 12862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:17.086476 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 12790
2026-06-21T02:50:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4470 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:17 dist-test-slave-bc52 krb5kdc[13394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:17 dist-test-slave-bc52 krb5kdc[13394](info): set up 2 sockets
Jun 21 02:50:17 dist-test-slave-bc52 krb5kdc[13394](info): commencing operation
krb5kdc: starting...
W20260621 02:50:18.173586 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:18 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:18.346532 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45907
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:18.651024 13407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:18.651297 13407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:18.651383 13407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:18.658259 13407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:18.658351 13407 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:50:18.658389 13407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:18.658449 13407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:18.658496 13407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:18.658553 13407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:18.667834 13407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45907
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:18.668941 13407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:18.670153 13407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:18.681463 13412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:18.681531 13415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:18.681562 13413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:18.682410 13407 server_base.cc:1034] running on GCE node
I20260621 02:50:18.683142 13407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:18.684495 13407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:18.685704 13407 hybrid_clock.cc:648] HybridClock initialized: now 1782010218685651 us; error 28 us; skew 500 ppm
I20260621 02:50:18.686024 13407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:18 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010218, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:18.688966 13407 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:18.690279 13407 webserver.cc:460] Webserver started at http://127.11.109.62:39271/ using document root <none> and password file <none>
I20260621 02:50:18.690901 13407 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:18.691027 13407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:18.691310 13407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:18.694195 13407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "d78f6afcfedc4ac69aa600839e635147"
format_stamp: "Formatted at 2026-06-21 02:50:18 on dist-test-slave-bc52"
server_key: "dd7f4fb9a6e4722cbb4231fae21a2895"
server_key_iv: "6e7b55b079fc08e5031c3b5a5f301bbd"
server_key_version: "encryptionkey@0"
I20260621 02:50:18.694913 13407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "d78f6afcfedc4ac69aa600839e635147"
format_stamp: "Formatted at 2026-06-21 02:50:18 on dist-test-slave-bc52"
server_key: "dd7f4fb9a6e4722cbb4231fae21a2895"
server_key_iv: "6e7b55b079fc08e5031c3b5a5f301bbd"
server_key_version: "encryptionkey@0"
I20260621 02:50:18.699697 13407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:50:18.703286 13422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:18.704530 13407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:18.704656 13407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "d78f6afcfedc4ac69aa600839e635147"
format_stamp: "Formatted at 2026-06-21 02:50:18 on dist-test-slave-bc52"
server_key: "dd7f4fb9a6e4722cbb4231fae21a2895"
server_key_iv: "6e7b55b079fc08e5031c3b5a5f301bbd"
server_key_version: "encryptionkey@0"
I20260621 02:50:18.704816 13407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:18.723477 13407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:18.727001 13407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:18.727193 13407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:18.747686 13407 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45907
I20260621 02:50:18.747702 13483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45907 every 8 connection(s)
I20260621 02:50:18.749017 13407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:18.752372 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 13407
I20260621 02:50:18.752562 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:18.752790 13484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:18.753321 11700 external_mini_cluster.cc:1428] Setting key f75565938cce580691681bd0c83002bf
I20260621 02:50:18.767206 13484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Bootstrap starting.
Jun 21 02:50:18 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:18.770820 13484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:18.772111 13484 log.cc:824] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:18.774797 13484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: No bootstrap required, opened a new log
I20260621 02:50:18.776927 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:18.755242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.11.109.62:45907)
0621 02:50:18.756108 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:18.756128 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:18.756174 (+ 46us) server_negotiation.cc:407] Connection header received
0621 02:50:18.758006 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:18.758033 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:18.758508 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:18.758918 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:18.760140 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:18.762073 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:18.763306 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:18.764373 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:18.764554 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:18.768837 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:18.768847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:18.768867 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:18.768912 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:18.772718 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:18.773488 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:18.773503 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:18.773508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:18.773669 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:18.774033 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:18.774038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:18.774041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:18.774510 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:18.774825 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:18.775217 (+ 392us) server_negotiation.cc:299] Negotiation successful
0621 02:50:18.775628 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":368,"threads_started":1}
I20260621 02:50:18.783818 13484 raft_consensus.cc:348] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78f6afcfedc4ac69aa600839e635147" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45907 } }
I20260621 02:50:18.784125 13484 raft_consensus.cc:374] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:18.784195 13484 raft_consensus.cc:729] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78f6afcfedc4ac69aa600839e635147, State: Initialized, Role: FOLLOWER
I20260621 02:50:18.784617 13484 consensus_queue.cc:260] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78f6afcfedc4ac69aa600839e635147" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45907 } }
I20260621 02:50:18.784855 13484 raft_consensus.cc:388] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:18.784950 13484 raft_consensus.cc:482] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:18.785063 13484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:18.787349 13484 raft_consensus.cc:504] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78f6afcfedc4ac69aa600839e635147" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45907 } }
I20260621 02:50:18.787847 13484 leader_election.cc:302] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d78f6afcfedc4ac69aa600839e635147; no voters:
I20260621 02:50:18.788337 13484 leader_election.cc:288] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:18.788738 13489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:18.789495 13489 raft_consensus.cc:686] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [term 1 LEADER]: Becoming Leader. State: Replica: d78f6afcfedc4ac69aa600839e635147, State: Running, Role: LEADER
I20260621 02:50:18.789986 13489 consensus_queue.cc:237] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78f6afcfedc4ac69aa600839e635147" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45907 } }
I20260621 02:50:18.790413 13484 sys_catalog.cc:564] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:18.792843 13491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d78f6afcfedc4ac69aa600839e635147. Latest consensus state: current_term: 1 leader_uuid: "d78f6afcfedc4ac69aa600839e635147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78f6afcfedc4ac69aa600839e635147" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45907 } } }
I20260621 02:50:18.792969 13490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d78f6afcfedc4ac69aa600839e635147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78f6afcfedc4ac69aa600839e635147" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45907 } } }
I20260621 02:50:18.793252 13491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:18.793361 13490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:18.796808 13496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:18.801777 13496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:18.810117 13496 catalog_manager.cc:1347] Generated new cluster ID: 98d24be7c6094b0a850b3ffaf9b9ad66
I20260621 02:50:18.810216 13496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:18.827579 13496 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:18.829350 13496 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:18.839390 13496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Generated new TSK 0
I20260621 02:50:18.840487 13496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:18.896600 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45907
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:19.210368 13510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:19.210650 13510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:19.210822 13510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:19.217584 13510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:19.217705 13510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:19.217846 13510 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:50:19.217927 13510 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:50:19.218060 13510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:19.227278 13510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:45907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:19.228426 13510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:19.229687 13510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:19.242990 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:19.243058 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:19.243079 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:19.244357 13510 server_base.cc:1034] running on GCE node
I20260621 02:50:19.244891 13510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:19.245777 13510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:19.247015 13510 hybrid_clock.cc:648] HybridClock initialized: now 1782010219246982 us; error 48 us; skew 500 ppm
I20260621 02:50:19.247303 13510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:19 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:19.250331 13510 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:19.251763 13510 webserver.cc:460] Webserver started at http://127.11.109.1:34835/ using document root <none> and password file <none>
I20260621 02:50:19.252553 13510 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:19.252707 13510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:19.253001 13510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:19.255823 13510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "f5267367f49444caa6c0881a96ef3c50"
format_stamp: "Formatted at 2026-06-21 02:50:19 on dist-test-slave-bc52"
server_key: "347aa6650ab9765959ee11de866df256"
server_key_iv: "e8f21c011594a1ea659acebf9b4f4e07"
server_key_version: "encryptionkey@0"
I20260621 02:50:19.256556 13510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "f5267367f49444caa6c0881a96ef3c50"
format_stamp: "Formatted at 2026-06-21 02:50:19 on dist-test-slave-bc52"
server_key: "347aa6650ab9765959ee11de866df256"
server_key_iv: "e8f21c011594a1ea659acebf9b4f4e07"
server_key_version: "encryptionkey@0"
I20260621 02:50:19.261480 13510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260621 02:50:19.264686 13525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:19.265945 13510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:19.266116 13510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f5267367f49444caa6c0881a96ef3c50"
format_stamp: "Formatted at 2026-06-21 02:50:19 on dist-test-slave-bc52"
server_key: "347aa6650ab9765959ee11de866df256"
server_key_iv: "e8f21c011594a1ea659acebf9b4f4e07"
server_key_version: "encryptionkey@0"
I20260621 02:50:19.266320 13510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:19.284561 13510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:19.288239 13510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:19.288509 13510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:19.291162 13510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:19.291332 13510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:19.291477 13510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:19.291570 13510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:50:19 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:19.314458 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.291618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:39789 (local address 127.11.109.62:45907)
0621 02:50:19.292658 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:19.292667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:19.293513 (+ 846us) server_negotiation.cc:407] Connection header received
0621 02:50:19.296565 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:19.296571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:19.296645 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:19.296765 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:19.299100 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.300031 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.302683 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.303207 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.303336 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.307601 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:19.307609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:19.307614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:19.307661 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:19.311133 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.312146 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.312151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.312155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.312283 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.312711 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.312716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.312719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.312973 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:19.313144 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:19.313882 (+ 738us) server_negotiation.cc:299] Negotiation successful
0621 02:50:19.314102 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":771,"threads_started":1}
I20260621 02:50:19.315696 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.292349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.11.109.1:39789)
0621 02:50:19.293337 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:19.293382 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:19.296211 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:19.297069 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:19.297078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:19.298014 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:19.298857 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.298876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.300187 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.300192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.302534 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.302545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.303436 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.303444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.303646 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.304399 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:19.304420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:19.307288 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:19.311307 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.311326 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.311340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.312018 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.312392 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.312397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.312400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.312595 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.313104 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:19.313111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:19.313519 (+ 408us) client_negotiation.cc:769] Sending connection context
0621 02:50:19.313790 (+ 271us) client_negotiation.cc:240] Negotiation successful
0621 02:50:19.314297 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":428,"threads_started":1}
I20260621 02:50:19.343353 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.321379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.11.109.1:53027)
0621 02:50:19.325910 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:19.325935 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:19.326153 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:19.326584 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:19.326592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:19.327441 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:19.328006 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.328023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.329602 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.329610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.331850 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.331870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.332991 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.333000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.333261 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.334072 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:19.334101 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:19.335590 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:19.339711 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.339718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.339724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.340459 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.340957 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.340964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.340970 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.341144 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.341917 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:19.341925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:19.342214 (+ 289us) client_negotiation.cc:769] Sending connection context
0621 02:50:19.342395 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 02:50:19.342861 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4343,"thread_start_us":4070,"threads_started":1}
I20260621 02:50:19.344465 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.321289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:53027 (local address 127.11.109.62:45907)
0621 02:50:19.321491 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:19.321499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:19.326023 (+ 4524us) server_negotiation.cc:407] Connection header received
0621 02:50:19.326310 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:19.326318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:19.326408 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:19.326520 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:19.328208 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.329411 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.332018 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.332762 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.332915 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.335832 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:19.335840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:19.335844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:19.335886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:19.339495 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.340608 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.340618 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.340624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.340804 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.341328 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.341334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.341337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.341728 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:19.343630 (+ 1902us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:19.343809 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 02:50:19.344066 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:50:19.344934 13510 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:41969
I20260621 02:50:19.344980 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:41969 every 8 connection(s)
I20260621 02:50:19.346264 13510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:19.348452 13646 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45907
I20260621 02:50:19.348685 13646 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:19.349355 13646 heartbeater.cc:511] Master 127.11.109.62:45907 requested a full tablet report, sending...
I20260621 02:50:19.351204 13439 ts_manager.cc:194] Registered new tserver with Master: f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969)
I20260621 02:50:19.352576 13439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:39789
I20260621 02:50:19.353992 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 13510
I20260621 02:50:19.354187 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:19.354782 11700 external_mini_cluster.cc:1428] Setting key 1e508c4f20935c7373c43bf4ac47d87c
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:19.391646 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45907
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:19.698773 13650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:19.699079 13650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:19.699239 13650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:19.706372 13650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:19.706504 13650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:19.706636 13650 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:50:19.706718 13650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:50:19.706840 13650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:19.716460 13650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:45907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:19.717787 13650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:19.719120 13650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:19.731503 13656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:19.731563 13658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:19.731515 13655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:19.732081 13650 server_base.cc:1034] running on GCE node
I20260621 02:50:19.732692 13650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:19.733590 13650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:19.734831 13650 hybrid_clock.cc:648] HybridClock initialized: now 1782010219734777 us; error 42 us; skew 500 ppm
I20260621 02:50:19.735128 13650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:19 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:19.738134 13650 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:19.739446 13650 webserver.cc:460] Webserver started at http://127.11.109.2:44851/ using document root <none> and password file <none>
I20260621 02:50:19.740061 13650 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:19.740217 13650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:19.740535 13650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:19.743340 13650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "573fba2d72cc452ebc17b628e4f6ac5d"
format_stamp: "Formatted at 2026-06-21 02:50:19 on dist-test-slave-bc52"
server_key: "75223df50c9313c3b724c12e394e9f2f"
server_key_iv: "b3ea5a6647c2f70d5740f7239668a32f"
server_key_version: "encryptionkey@0"
I20260621 02:50:19.744079 13650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "573fba2d72cc452ebc17b628e4f6ac5d"
format_stamp: "Formatted at 2026-06-21 02:50:19 on dist-test-slave-bc52"
server_key: "75223df50c9313c3b724c12e394e9f2f"
server_key_iv: "b3ea5a6647c2f70d5740f7239668a32f"
server_key_version: "encryptionkey@0"
I20260621 02:50:19.749297 13650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:50:19.752678 13665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:19.753996 13650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:19.754173 13650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "573fba2d72cc452ebc17b628e4f6ac5d"
format_stamp: "Formatted at 2026-06-21 02:50:19 on dist-test-slave-bc52"
server_key: "75223df50c9313c3b724c12e394e9f2f"
server_key_iv: "b3ea5a6647c2f70d5740f7239668a32f"
server_key_version: "encryptionkey@0"
I20260621 02:50:19.754364 13650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:19.774456 13650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:19.778528 13650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:19.778826 13650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:19.787210 13650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:19.787351 13650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:19.787501 13650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:19.787611 13650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:50:19 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010219, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:19.807720 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.786060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:33619 (local address 127.11.109.62:45907)
0621 02:50:19.786288 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:19.786318 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:19.786338 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:50:19.788872 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:19.788878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:19.788961 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:19.789079 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:19.790606 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.791438 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.794889 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.795502 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.795631 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.800649 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:19.800659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:19.800666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:19.800719 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:19.804042 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.804896 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.804904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.804911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.805100 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.805548 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.805556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.805561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.806021 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:19.806209 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:19.806910 (+ 701us) server_negotiation.cc:299] Negotiation successful
0621 02:50:19.807245 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:50:19.808357 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.784697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.11.109.2:33619)
0621 02:50:19.786168 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:19.786220 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:19.788561 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:19.789336 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:19.789343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:19.789860 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:19.790442 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.790458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.791639 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.791650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.794709 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.794730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.795796 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.795810 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.796203 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.797006 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:19.797049 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:19.800345 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:19.804187 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.804195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.804212 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.804746 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.805240 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.805248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.805252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.805427 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.806172 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:19.806177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:19.806489 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 02:50:19.806719 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 02:50:19.807073 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":418,"threads_started":1}
I20260621 02:50:19.828877 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.815198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:34111 (local address 127.11.109.62:45907)
0621 02:50:19.815428 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:19.815434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:19.815915 (+ 481us) server_negotiation.cc:407] Connection header received
0621 02:50:19.816042 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:19.816049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:19.816143 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:19.816267 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:19.817234 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.818506 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.820700 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.821300 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.821435 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.823207 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:19.823219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:19.823223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:19.823250 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:19.825781 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.826657 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.826664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.826670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.826833 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:19.827339 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:19.827344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:19.827347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:19.827751 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:19.827854 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:19.828381 (+ 527us) server_negotiation.cc:299] Negotiation successful
0621 02:50:19.828537 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 02:50:19.829309 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:19.815212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.11.109.2:34111)
0621 02:50:19.815817 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:19.815837 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:19.815936 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:19.816276 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:19.816279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:19.816730 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:19.817069 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.817078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.818671 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.818683 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.820492 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:19.820513 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:19.821487 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:19.821493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:19.821653 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:19.822300 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:19.822316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:19.823010 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:19.825940 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.825944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.825946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.826522 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.826994 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:19.827003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:19.827008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:19.827195 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:19.827882 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:19.827893 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:19.828125 (+ 232us) client_negotiation.cc:769] Sending connection context
0621 02:50:19.828309 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 02:50:19.828651 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":335,"threads_started":1}
I20260621 02:50:19.841658 13650 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:40815
I20260621 02:50:19.841701 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:40815 every 8 connection(s)
I20260621 02:50:19.842862 13650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:19.844964 13785 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45907
I20260621 02:50:19.845261 13785 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:19.845964 13785 heartbeater.cc:511] Master 127.11.109.62:45907 requested a full tablet report, sending...
I20260621 02:50:19.847190 13439 ts_manager.cc:194] Registered new tserver with Master: 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815)
I20260621 02:50:19.848065 13439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:33619
I20260621 02:50:19.848165 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 13650
I20260621 02:50:19.848313 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:19.848847 11700 external_mini_cluster.cc:1428] Setting key 5f0817df26b939e99d0eeb041364b505
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:19.888614 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45907
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:20.197948 13789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:20.198197 13789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:20.198385 13789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:20.205423 13789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:20.205528 13789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:20.205699 13789 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:50:20.205776 13789 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:50:20.205901 13789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:20.215190 13789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:45907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:20.216361 13789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:20.217849 13789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:20.230449 13794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:20.230513 13795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:20.230470 13797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:20.231406 13789 server_base.cc:1034] running on GCE node
I20260621 02:50:20.231976 13789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:20.232844 13789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:20.234104 13789 hybrid_clock.cc:648] HybridClock initialized: now 1782010220234051 us; error 54 us; skew 500 ppm
I20260621 02:50:20.234530 13789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:20 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010220, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:20.238300 13789 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:20.239734 13789 webserver.cc:460] Webserver started at http://127.11.109.3:44377/ using document root <none> and password file <none>
I20260621 02:50:20.240500 13789 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:20.240650 13789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:20.240955 13789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:20.243758 13789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
format_stamp: "Formatted at 2026-06-21 02:50:20 on dist-test-slave-bc52"
server_key: "15d8e85da987f8bbf3c078ab6ea847c3"
server_key_iv: "ca23db32bc6d5879ac11483dd1c589a6"
server_key_version: "encryptionkey@0"
I20260621 02:50:20.244501 13789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
format_stamp: "Formatted at 2026-06-21 02:50:20 on dist-test-slave-bc52"
server_key: "15d8e85da987f8bbf3c078ab6ea847c3"
server_key_iv: "ca23db32bc6d5879ac11483dd1c589a6"
server_key_version: "encryptionkey@0"
I20260621 02:50:20.250732 13789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260621 02:50:20.255277 13804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:20.256531 13789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:20.256692 13789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
format_stamp: "Formatted at 2026-06-21 02:50:20 on dist-test-slave-bc52"
server_key: "15d8e85da987f8bbf3c078ab6ea847c3"
server_key_iv: "ca23db32bc6d5879ac11483dd1c589a6"
server_key_version: "encryptionkey@0"
I20260621 02:50:20.256888 13789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:20.278513 13789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:20.282305 13789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:20.282575 13789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:20.285151 13789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:20.285284 13789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:20.285411 13789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:20.285516 13789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:50:20 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010220, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:20.307509 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.285765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:37377 (local address 127.11.109.62:45907)
0621 02:50:20.285983 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.285990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.287296 (+ 1306us) server_negotiation.cc:407] Connection header received
0621 02:50:20.289230 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.289236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.289321 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.289445 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:20.291679 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.292822 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.295442 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.296071 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.296237 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.299935 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:20.299940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:20.299943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:20.299973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:20.303520 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.304577 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.304584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.304590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.304777 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.305303 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.305310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.305313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.305586 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:20.305770 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.306675 (+ 905us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.306984 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:50:20.308087 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.286370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.11.109.3:37377)
0621 02:50:20.287139 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.287182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.288936 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.289679 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.289686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.290454 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:20.291429 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.291457 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.292967 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.292980 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.295294 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.295305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.296322 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.296329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.296596 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.297306 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:20.297327 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:20.299729 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:20.303667 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:20.303678 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:20.303687 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:20.304440 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:20.304911 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:20.304917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:20.304922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:20.305146 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:20.305744 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:20.305750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:20.306235 (+ 485us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.306539 (+ 304us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.306941 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":327,"threads_started":1}
I20260621 02:50:20.327768 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.313271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.11.109.3:32835)
0621 02:50:20.313971 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.313990 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.314086 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.314421 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.314424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.314795 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:20.315059 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.315066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.316683 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.316687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.318047 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.318058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.319008 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.319017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.319269 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.319943 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:20.319972 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:20.321301 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:20.325163 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:20.325192 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:20.325196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:20.325682 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:20.326162 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:20.326168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:20.326172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:20.326291 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:20.326977 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:20.326982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:20.327133 (+ 151us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.327247 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.327467 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":430,"threads_started":1}
I20260621 02:50:20.328244 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.313283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:32835 (local address 127.11.109.62:45907)
0621 02:50:20.313454 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.313462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.314074 (+ 612us) server_negotiation.cc:407] Connection header received
0621 02:50:20.314190 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.314199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.314290 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.314418 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:20.315198 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.316544 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.318215 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.318756 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.318884 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.321500 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:20.321507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:20.321513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:20.321551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:20.325031 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.325828 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.325835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.325842 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.326036 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.326411 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.326419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.326425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.326840 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:20.327034 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.327407 (+ 373us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.327739 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:50:20.345726 13789 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:42337
I20260621 02:50:20.345745 13923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:42337 every 8 connection(s)
I20260621 02:50:20.346942 13789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:20.348946 13924 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45907
I20260621 02:50:20.349202 13924 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:20.349890 13924 heartbeater.cc:511] Master 127.11.109.62:45907 requested a full tablet report, sending...
I20260621 02:50:20.351159 13439 ts_manager.cc:194] Registered new tserver with Master: f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:20.351953 13439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:37377
I20260621 02:50:20.355147 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 13789
I20260621 02:50:20.355331 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:20.355310 13646 heartbeater.cc:503] Master 127.11.109.62:45907 was elected leader, sending a full tablet report...
I20260621 02:50:20.355818 11700 external_mini_cluster.cc:1428] Setting key 3ff2c27783add291d9ea528144826de9
I20260621 02:50:20.358685 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:50:20 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:50:20 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:20.406332 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.390353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.11.109.62:45907)
0621 02:50:20.390530 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.390534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.390871 (+ 337us) server_negotiation.cc:407] Connection header received
0621 02:50:20.390984 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.390988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.391121 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.391212 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:20.392606 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.393412 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.394592 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.394966 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.395052 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.401817 (+ 6765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:20.401822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:20.401825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:20.401857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:20.404141 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.404731 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.404735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.404737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.404843 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.405162 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.405165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.405167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.405464 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:20.405650 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.405894 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.406044 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:50:20.410727 13439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:20.417311 13439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:20.440179 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.429506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.11.109.2:40815)
0621 02:50:20.430130 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.430137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.430675 (+ 538us) server_negotiation.cc:407] Connection header received
0621 02:50:20.430895 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.430903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.431187 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.431381 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.432446 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.434464 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.437641 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.438949 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.439109 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.439319 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.439724 (+ 405us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.439909 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":314,"threads_started":1}
I20260621 02:50:20.441156 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.429357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:40815 (local address 127.0.0.1:51204)
0621 02:50:20.430528 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.430565 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.430707 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.431625 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.431635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.431707 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.432190 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.432206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.434658 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.434666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.437492 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.437503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.439194 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.439201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.439409 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.439458 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.440419 (+ 961us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.440679 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"spinlock_wait_cycles":33536,"thread_start_us":886,"threads_started":1}
I20260621 02:50:20.443301 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.431503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:42337 (local address 127.0.0.1:35524)
0621 02:50:20.435108 (+ 3605us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.435123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.435207 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.436236 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.436240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.436272 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.436718 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.436730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.438824 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.438829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.440810 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.440823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.442420 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.442426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.442660 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.442703 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.442771 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.442881 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3497,"mutex_wait_us":1,"thread_start_us":3337,"threads_started":1}
I20260621 02:50:20.444902 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.430934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41969 (local address 127.0.0.1:60898)
0621 02:50:20.431521 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.431544 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.431707 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.437931 (+ 6224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.437935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.437960 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.438323 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.438334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.440528 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.440533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.442715 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.442727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.444097 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.444121 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.444294 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.444339 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.444427 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.444544 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":47,"spinlock_wait_cycles":2432,"thread_start_us":255,"threads_started":1}
I20260621 02:50:20.444969 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.431200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.11.109.1:41969)
0621 02:50:20.437371 (+ 6171us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.437380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.437410 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:50:20.437505 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.437513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.437739 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.437927 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.438496 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.440322 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.442860 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.443900 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.444052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.444260 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.444503 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.444669 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6004,"thread_start_us":5837,"threads_started":1}
I20260621 02:50:20.445209 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.432047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.11.109.3:42337)
0621 02:50:20.435669 (+ 3622us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.435675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.435699 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:50:20.435767 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.435772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.436029 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.436901 (+ 872us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.437266 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.438669 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.441001 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.442168 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.444383 (+ 2215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.444617 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.444684 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.444837 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3489,"thread_start_us":3356,"threads_started":1}
I20260621 02:50:20.445013 13715 tablet_service.cc:1467] Processing CreateTablet for tablet c8e13468cea94a6496e1baf9b4f56c65 (DEFAULT_TABLE table=test-table [id=d16d6c0324b549fbadf34c0defe1cb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:20.447918 13576 tablet_service.cc:1467] Processing CreateTablet for tablet c8e13468cea94a6496e1baf9b4f56c65 (DEFAULT_TABLE table=test-table [id=d16d6c0324b549fbadf34c0defe1cb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:20.447940 13859 tablet_service.cc:1467] Processing CreateTablet for tablet c8e13468cea94a6496e1baf9b4f56c65 (DEFAULT_TABLE table=test-table [id=d16d6c0324b549fbadf34c0defe1cb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:20.448886 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e13468cea94a6496e1baf9b4f56c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:20.448999 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e13468cea94a6496e1baf9b4f56c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:20.449065 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e13468cea94a6496e1baf9b4f56c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:20.460063 13943 tablet_bootstrap.cc:492] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Bootstrap starting.
I20260621 02:50:20.462816 13944 tablet_bootstrap.cc:492] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Bootstrap starting.
I20260621 02:50:20.463696 13943 tablet_bootstrap.cc:654] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:20.464363 13945 tablet_bootstrap.cc:492] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Bootstrap starting.
I20260621 02:50:20.465062 13943 log.cc:824] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:20.466620 13944 tablet_bootstrap.cc:654] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:20.467363 13945 tablet_bootstrap.cc:654] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:20.467809 13944 log.cc:824] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:20.468190 13943 tablet_bootstrap.cc:492] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: No bootstrap required, opened a new log
I20260621 02:50:20.468456 13943 ts_tablet_manager.cc:1397] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:50:20.468503 13945 log.cc:824] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:20.470862 13944 tablet_bootstrap.cc:492] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: No bootstrap required, opened a new log
I20260621 02:50:20.471118 13944 ts_tablet_manager.cc:1397] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:50:20.471285 13945 tablet_bootstrap.cc:492] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: No bootstrap required, opened a new log
I20260621 02:50:20.471540 13945 ts_tablet_manager.cc:1397] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:50:20.478029 13944 raft_consensus.cc:348] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.478463 13944 raft_consensus.cc:374] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:20.478559 13944 raft_consensus.cc:729] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5267367f49444caa6c0881a96ef3c50, State: Initialized, Role: FOLLOWER
I20260621 02:50:20.478965 13943 raft_consensus.cc:348] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.479161 13944 consensus_queue.cc:260] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.479341 13943 raft_consensus.cc:374] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:20.479427 13943 raft_consensus.cc:729] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f731a284dd5a4959bbf9f3d2b90d7faf, State: Initialized, Role: FOLLOWER
I20260621 02:50:20.479990 13943 consensus_queue.cc:260] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.481086 13924 heartbeater.cc:503] Master 127.11.109.62:45907 was elected leader, sending a full tablet report...
I20260621 02:50:20.481812 13943 ts_tablet_manager.cc:1428] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260621 02:50:20.481983 13945 raft_consensus.cc:348] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.482383 13945 raft_consensus.cc:374] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:20.482489 13945 raft_consensus.cc:729] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Initialized, Role: FOLLOWER
I20260621 02:50:20.483122 13944 ts_tablet_manager.cc:1428] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260621 02:50:20.483027 13945 consensus_queue.cc:260] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.484102 13785 heartbeater.cc:503] Master 127.11.109.62:45907 was elected leader, sending a full tablet report...
I20260621 02:50:20.485342 13945 ts_tablet_manager.cc:1428] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260621 02:50:20.530689 13951 raft_consensus.cc:482] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:20.530969 13951 raft_consensus.cc:504] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.532929 13951 leader_election.cc:288] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:20.539158 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.533310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:55719 (local address 127.11.109.1:41969)
0621 02:50:20.533502 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.533506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.533906 (+ 400us) server_negotiation.cc:407] Connection header received
0621 02:50:20.534231 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.534236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.534331 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.534399 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.535017 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.535712 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.537503 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.538245 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.538427 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.538499 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.538795 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.538916 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:50:20.539414 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.533114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41969 (local address 127.11.109.2:55719)
0621 02:50:20.533804 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.533828 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.534026 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.534553 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.534556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.534581 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.534882 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.534890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.535841 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.535848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.537343 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.537354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.538450 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.538456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.538627 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.538650 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.538772 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.538895 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260621 02:50:20.539463 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.534086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:58643 (local address 127.11.109.3:42337)
0621 02:50:20.534292 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.534300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.534557 (+ 257us) server_negotiation.cc:407] Connection header received
0621 02:50:20.534761 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.534767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.534837 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.534927 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.535449 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.536132 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.537818 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.538580 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.538762 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.538831 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.539106 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.539224 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:50:20.539542 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.533867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:42337 (local address 127.11.109.2:58643)
0621 02:50:20.534437 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.534450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.534621 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.534940 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.534944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.534967 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.535307 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.535315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.536267 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.536272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.537675 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.537687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.538789 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.538795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.538950 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.538972 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.539072 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.539193 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":278,"threads_started":1}
I20260621 02:50:20.540486 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e13468cea94a6496e1baf9b4f56c65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50" is_pre_election: true
I20260621 02:50:20.540503 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e13468cea94a6496e1baf9b4f56c65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" is_pre_election: true
I20260621 02:50:20.540948 13879 raft_consensus.cc:2454] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 0.
I20260621 02:50:20.540953 13601 raft_consensus.cc:2454] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 0.
I20260621 02:50:20.541652 13668 leader_election.cc:302] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f5267367f49444caa6c0881a96ef3c50; no voters:
I20260621 02:50:20.542047 13951 raft_consensus.cc:2781] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:20.542182 13951 raft_consensus.cc:482] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:20.542305 13951 raft_consensus.cc:3037] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:20.544400 13951 raft_consensus.cc:504] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.544970 13951 leader_election.cc:288] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 election: Requested vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:20.545413 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e13468cea94a6496e1baf9b4f56c65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50"
I20260621 02:50:20.545477 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e13468cea94a6496e1baf9b4f56c65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
I20260621 02:50:20.545626 13601 raft_consensus.cc:3037] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:20.545687 13879 raft_consensus.cc:3037] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:20.547721 13601 raft_consensus.cc:2454] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 1.
I20260621 02:50:20.547781 13879 raft_consensus.cc:2454] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 1.
I20260621 02:50:20.548135 13668 leader_election.cc:302] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f5267367f49444caa6c0881a96ef3c50; no voters:
I20260621 02:50:20.548447 13951 raft_consensus.cc:2781] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:20.548887 13951 raft_consensus.cc:686] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 LEADER]: Becoming Leader. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Running, Role: LEADER
I20260621 02:50:20.549422 13951 consensus_queue.cc:237] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.554148 13437 catalog_manager.cc:5495] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2). New cstate: current_term: 1 leader_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } health_report { overall_health: UNKNOWN } } }
W20260621 02:50:20.594753 13786 tablet.cc:2369] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:50:20.598433 13925 tablet.cc:2369] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:50:20.599221 13647 tablet.cc:2369] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:50:20 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010218, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:20.620359 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.607364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.11.109.62:45907)
0621 02:50:20.607514 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.607518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.608137 (+ 619us) server_negotiation.cc:407] Connection header received
0621 02:50:20.608345 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.608349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.608424 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.608527 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:20.609800 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.610511 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.611743 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.612174 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.612308 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.615414 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:20.615424 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:20.615427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:20.615453 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:20.617778 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.618444 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.618450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.618454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.618576 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:20.618944 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:20.618951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:20.618954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:20.619213 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:20.619358 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.619751 (+ 393us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.620066 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":47}
I20260621 02:50:20.620396 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.607467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45907 (local address 127.0.0.1:33194)
0621 02:50:20.608033 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.608048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.608207 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.608569 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.608573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.609398 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:20.609667 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.609675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.610660 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.610668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.611607 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.611619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.612345 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.612351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.612522 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.613085 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:20.613115 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:20.615250 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:20.617899 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:20.617912 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:20.617937 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:20.618320 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:20.618691 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:20.618696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:20.618700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:20.618833 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:20.619333 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:20.619340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:20.619510 (+ 170us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.619666 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.619937 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":272,"threads_started":1}
I20260621 02:50:20.625304 13437 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:33194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:50:20.626348 13437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:20.632938 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 4d18a2435b7b4926b854a4957ec410bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0cd03d903da4417a87a87ab1a78e819f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:50:20.632937 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 4d18a2435b7b4926b854a4957ec410bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0cd03d903da4417a87a87ab1a78e819f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:50:20.633212 13859 tablet_service.cc:1467] Processing CreateTablet for tablet 4d18a2435b7b4926b854a4957ec410bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0cd03d903da4417a87a87ab1a78e819f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:50:20.633904 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d18a2435b7b4926b854a4957ec410bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:20.633903 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d18a2435b7b4926b854a4957ec410bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:20.633931 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d18a2435b7b4926b854a4957ec410bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:20.639528 13944 tablet_bootstrap.cc:492] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: Bootstrap starting.
I20260621 02:50:20.640838 13945 tablet_bootstrap.cc:492] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: Bootstrap starting.
I20260621 02:50:20.641068 13944 tablet_bootstrap.cc:654] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:20.641268 13943 tablet_bootstrap.cc:492] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: Bootstrap starting.
I20260621 02:50:20.642553 13945 tablet_bootstrap.cc:654] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:20.642913 13943 tablet_bootstrap.cc:654] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:20.643465 13944 tablet_bootstrap.cc:492] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: No bootstrap required, opened a new log
I20260621 02:50:20.643635 13944 ts_tablet_manager.cc:1397] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:50:20.644258 13944 raft_consensus.cc:348] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.644505 13944 raft_consensus.cc:374] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:20.644606 13944 raft_consensus.cc:729] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5267367f49444caa6c0881a96ef3c50, State: Initialized, Role: FOLLOWER
I20260621 02:50:20.644979 13945 tablet_bootstrap.cc:492] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: No bootstrap required, opened a new log
I20260621 02:50:20.644825 13944 consensus_queue.cc:260] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.645562 13944 ts_tablet_manager.cc:1428] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:20.645643 13943 tablet_bootstrap.cc:492] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: No bootstrap required, opened a new log
I20260621 02:50:20.645341 13949 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:20.645773 13949 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:20.645776 13943 ts_tablet_manager.cc:1397] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 02:50:20.645776 13945 ts_tablet_manager.cc:1397] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 02:50:20.646503 13945 raft_consensus.cc:348] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.646559 13943 raft_consensus.cc:348] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.646751 13945 raft_consensus.cc:374] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:20.646801 13943 raft_consensus.cc:374] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:20.646837 13945 raft_consensus.cc:729] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Initialized, Role: FOLLOWER
I20260621 02:50:20.646906 13943 raft_consensus.cc:729] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f731a284dd5a4959bbf9f3d2b90d7faf, State: Initialized, Role: FOLLOWER
I20260621 02:50:20.647084 13945 consensus_queue.cc:260] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.647146 13943 consensus_queue.cc:260] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.647504 13951 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:20.647809 13951 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:20.648239 13945 ts_tablet_manager.cc:1428] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:20.648161 13950 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:20.648512 13950 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:20.649139 13943 ts_tablet_manager.cc:1428] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:20.931032 13950 raft_consensus.cc:482] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:20.931247 13950 raft_consensus.cc:504] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.933089 13950 leader_election.cc:288] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815)
I20260621 02:50:20.939481 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.933296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41969 (local address 127.11.109.3:52439)
0621 02:50:20.933762 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.933777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.933945 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.934521 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.934526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.934559 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.934789 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.934801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.936099 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.936104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.937516 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.937527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.938707 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.938713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.938905 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.938969 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.939040 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.939141 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"thread_start_us":249,"threads_started":1}
I20260621 02:50:20.939504 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.933408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:52439 (local address 127.11.109.1:41969)
0621 02:50:20.933626 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.933633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.933895 (+ 262us) server_negotiation.cc:407] Connection header received
0621 02:50:20.934123 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.934131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.934206 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.934269 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.935018 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.935946 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.937673 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.938529 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.938655 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.938705 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.939099 (+ 394us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.939226 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:50:20.939741 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.934085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:55505 (local address 127.11.109.2:40815)
0621 02:50:20.934282 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:20.934287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:20.934466 (+ 179us) server_negotiation.cc:407] Connection header received
0621 02:50:20.934593 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:20.934598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:20.934666 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:20.934766 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:20.935369 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.936183 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.937921 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.938697 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.938856 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.938924 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:20.939341 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 02:50:20.939481 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:50:20.939836 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:20.933872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:40815 (local address 127.11.109.3:55505)
0621 02:50:20.934356 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:20.934370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:20.934486 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:20.934808 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:20.934812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:20.934838 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:20.935210 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.935224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.936314 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.936317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.937770 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:20.937783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:20.938915 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:20.938919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:20.939175 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:20.939207 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:50:20.939297 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 02:50:20.939426 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":33,"thread_start_us":231,"threads_started":1}
I20260621 02:50:20.940514 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d18a2435b7b4926b854a4957ec410bc" candidate_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50" is_pre_election: true
I20260621 02:50:20.940761 13601 raft_consensus.cc:2454] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f731a284dd5a4959bbf9f3d2b90d7faf in term 0.
I20260621 02:50:20.940747 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d18a2435b7b4926b854a4957ec410bc" candidate_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" is_pre_election: true
I20260621 02:50:20.941226 13740 raft_consensus.cc:2454] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f731a284dd5a4959bbf9f3d2b90d7faf in term 0.
I20260621 02:50:20.941213 13807 leader_election.cc:302] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f5267367f49444caa6c0881a96ef3c50, f731a284dd5a4959bbf9f3d2b90d7faf; no voters:
I20260621 02:50:20.941628 13950 raft_consensus.cc:2781] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:20.941763 13950 raft_consensus.cc:482] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:20.941855 13950 raft_consensus.cc:3037] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:20.943917 13950 raft_consensus.cc:504] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.944443 13950 leader_election.cc:288] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [CANDIDATE]: Term 1 election: Requested vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815)
I20260621 02:50:20.944870 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d18a2435b7b4926b854a4957ec410bc" candidate_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50"
I20260621 02:50:20.944921 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d18a2435b7b4926b854a4957ec410bc" candidate_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573fba2d72cc452ebc17b628e4f6ac5d"
I20260621 02:50:20.945092 13740 raft_consensus.cc:3037] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:20.945097 13601 raft_consensus.cc:3037] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:20.946991 13601 raft_consensus.cc:2454] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f731a284dd5a4959bbf9f3d2b90d7faf in term 1.
I20260621 02:50:20.947250 13740 raft_consensus.cc:2454] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f731a284dd5a4959bbf9f3d2b90d7faf in term 1.
I20260621 02:50:20.947403 13807 leader_election.cc:302] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f5267367f49444caa6c0881a96ef3c50, f731a284dd5a4959bbf9f3d2b90d7faf; no voters:
I20260621 02:50:20.947710 13950 raft_consensus.cc:2781] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:20.948089 13950 raft_consensus.cc:686] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 LEADER]: Becoming Leader. State: Replica: f731a284dd5a4959bbf9f3d2b90d7faf, State: Running, Role: LEADER
I20260621 02:50:20.948565 13950 consensus_queue.cc:237] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:20.951617 13967 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: TxnStatusTablet state changed. Reason: New leader f731a284dd5a4959bbf9f3d2b90d7faf. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:20.951925 13967 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:50:20.952597 13969 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:50:20.952888 13969 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:50:20.953553 13439 catalog_manager.cc:5495] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf reported cstate change: term changed from 0 to 1, leader changed from <none> to f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3). New cstate: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } health_report { overall_health: HEALTHY } } }
I20260621 02:50:21.008867 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.003350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:42337 (local address 127.0.0.1:35526)
0621 02:50:21.003572 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.003589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.003724 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.004222 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.004226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.004237 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:21.004410 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.004418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.005493 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.005500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.006435 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.006446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.007169 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.007174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.007316 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.007368 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.008289 (+ 921us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.008308 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.008401 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.008547 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:50:21.008836 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.003649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.11.109.3:42337)
0621 02:50:21.003856 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.003864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.003882 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:50:21.003965 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.003969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.004037 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.004178 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.004534 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.005373 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.006584 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.006989 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.007154 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.007547 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.008172 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.008357 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.008416 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.008494 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:50:21.015566 13951 consensus_queue.cc:1035] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:21.017289 13740 raft_consensus.cc:1261] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Refusing update from remote peer f731a284dd5a4959bbf9f3d2b90d7faf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:21.017390 13600 raft_consensus.cc:1261] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Refusing update from remote peer f731a284dd5a4959bbf9f3d2b90d7faf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:21.018272 13967 consensus_queue.cc:1035] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [LEADER]: Connected to new peer: Peer: permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:21.018599 13950 consensus_queue.cc:1035] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:21.030527 13951 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: TxnStatusTablet state changed. Reason: New leader f731a284dd5a4959bbf9f3d2b90d7faf. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:21.030835 13951 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:21.033561 13951 consensus_queue.cc:1035] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:21.034412 13949 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: TxnStatusTablet state changed. Reason: New leader f731a284dd5a4959bbf9f3d2b90d7faf. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:21.034711 13949 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:21.038103 13972 mvcc.cc:205] Tried to move back new op lower bound from 7299113865277890560 to 7299113865009008640. Current Snapshot: MvccSnapshot[applied={T|T < 7299113865277890560}]
I20260621 02:50:21.050590 13967 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:21.050848 13967 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:50:21.053982 13975 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:21.054234 13975 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P 573fba2d72cc452ebc17b628e4f6ac5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:21.056807 13967 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:21.057049 13967 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f731a284dd5a4959bbf9f3d2b90d7faf: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:50:21.057590 13980 mvcc.cc:205] Tried to move back new op lower bound from 7299113865277890560 to 7299113865009008640. Current Snapshot: MvccSnapshot[applied={T|T < 7299113865277890560}]
I20260621 02:50:21.060678 13949 tablet_replica.cc:439] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } } }
I20260621 02:50:21.060920 13949 tablet_replica.cc:442] T 4d18a2435b7b4926b854a4957ec410bc P f5267367f49444caa6c0881a96ef3c50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:50:21.070008 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.063878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.11.109.2:40815)
0621 02:50:21.064054 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.064060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.064457 (+ 397us) server_negotiation.cc:407] Connection header received
0621 02:50:21.064633 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.064639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.064731 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.064880 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.065248 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.066345 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.067676 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.068292 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.068405 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.068739 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.069364 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.069501 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.069652 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.069750 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:50:21.085366 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.079488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:33831 (local address 127.11.109.3:42337)
0621 02:50:21.079705 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.079712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.079989 (+ 277us) server_negotiation.cc:407] Connection header received
0621 02:50:21.080186 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.080191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.080280 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.080405 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.080939 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.082099 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.083387 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.084077 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.084207 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.084563 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.084711 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.084830 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.084987 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.085061 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 02:50:21.085386 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.079386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:42337 (local address 127.11.109.2:33831)
0621 02:50:21.079892 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.079911 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.080059 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.080416 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.080421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.080444 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:21.080772 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.080784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.082216 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.082220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.083251 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.083262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.084234 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.084238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.084343 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.084389 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.084872 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.084880 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.084941 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.085032 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260621 02:50:21.123373 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.116823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:40815 (local address 127.11.109.3:32977)
0621 02:50:21.117447 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.117462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.117623 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.118094 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.118100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.118112 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:21.118532 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.118540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.119805 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.119810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.121153 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.121165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.122036 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.122044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.122208 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.122296 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.122728 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.122740 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.122835 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.122975 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":256,"threads_started":1}
I20260621 02:50:21.123226 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.117060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:32977 (local address 127.11.109.2:40815)
0621 02:50:21.117288 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.117294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.117543 (+ 249us) server_negotiation.cc:407] Connection header received
0621 02:50:21.117756 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.117761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.117841 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.117935 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.118805 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.119679 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.121343 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.121861 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.121998 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.122486 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.122621 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.122713 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.122869 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.122947 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:50:21.155400 13439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:21.155699 13439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:21.159590 13439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Sending DeleteTablet for 3 replicas of tablet c8e13468cea94a6496e1baf9b4f56c65
I20260621 02:50:21.161811 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e13468cea94a6496e1baf9b4f56c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:21 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51204
I20260621 02:50:21.161106 13576 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e13468cea94a6496e1baf9b4f56c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:21 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60898
I20260621 02:50:21.162739 13994 tablet_replica.cc:330] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: stopping tablet replica
I20260621 02:50:21.162889 13439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 02:50:21.163336 13994 raft_consensus.cc:2229] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 LEADER]: Raft consensus shutting down.
W20260621 02:50:21.163556 13439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:21.164052 13994 raft_consensus.cc:2258] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:21.165890 13994 ts_tablet_manager.cc:1905] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:21.172366 13858 tablet_service.cc:1467] Processing CreateTablet for tablet 6000394e714641128fab0f920e60fb12 (DEFAULT_TABLE table=test-table [id=4fb3cdcb105a416ebd7686ee37416265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:21.172864 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6000394e714641128fab0f920e60fb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:21.173974 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 6000394e714641128fab0f920e60fb12 (DEFAULT_TABLE table=test-table [id=4fb3cdcb105a416ebd7686ee37416265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:21.174037 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 6000394e714641128fab0f920e60fb12 (DEFAULT_TABLE table=test-table [id=4fb3cdcb105a416ebd7686ee37416265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:21.174387 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6000394e714641128fab0f920e60fb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:21.174515 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6000394e714641128fab0f920e60fb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:21.189051 13996 tablet_bootstrap.cc:492] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Bootstrap starting.
I20260621 02:50:21.188585 13859 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e13468cea94a6496e1baf9b4f56c65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:21 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:35524
I20260621 02:50:21.190877 13996 tablet_bootstrap.cc:654] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:21.196280 13997 tablet_replica.cc:330] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: stopping tablet replica
I20260621 02:50:21.196785 13997 raft_consensus.cc:2229] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:21.197247 13997 raft_consensus.cc:2258] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:21.199297 13997 ts_tablet_manager.cc:1905] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:21.174036 13994 ts_tablet_manager.cc:1918] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:21.207196 14000 tablet_bootstrap.cc:492] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Bootstrap starting.
I20260621 02:50:21.209470 14000 tablet_bootstrap.cc:654] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:21.210855 13994 log.cc:1196] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/c8e13468cea94a6496e1baf9b4f56c65
I20260621 02:50:21.211639 13994 ts_tablet_manager.cc:1939] T c8e13468cea94a6496e1baf9b4f56c65 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting consensus metadata
I20260621 02:50:21.213762 13995 tablet_replica.cc:330] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: stopping tablet replica
I20260621 02:50:21.214186 13995 raft_consensus.cc:2229] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:21.214416 13996 tablet_bootstrap.cc:492] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: No bootstrap required, opened a new log
I20260621 02:50:21.214564 13995 raft_consensus.cc:2258] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:21.215402 13425 catalog_manager.cc:4841] TS 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815): tablet c8e13468cea94a6496e1baf9b4f56c65 (table test-table [id=d16d6c0324b549fbadf34c0defe1cb40]) successfully deleted
I20260621 02:50:21.214562 13996 ts_tablet_manager.cc:1397] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.012s
I20260621 02:50:21.216382 13999 tablet_bootstrap.cc:492] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Bootstrap starting.
I20260621 02:50:21.216416 13997 ts_tablet_manager.cc:1918] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:21.216377 13996 raft_consensus.cc:348] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.216576 13997 log.cc:1196] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/c8e13468cea94a6496e1baf9b4f56c65
I20260621 02:50:21.216606 13996 raft_consensus.cc:374] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:21.216714 13996 raft_consensus.cc:729] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5267367f49444caa6c0881a96ef3c50, State: Initialized, Role: FOLLOWER
I20260621 02:50:21.217109 13997 ts_tablet_manager.cc:1939] T c8e13468cea94a6496e1baf9b4f56c65 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting consensus metadata
I20260621 02:50:21.216961 13996 consensus_queue.cc:260] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.220037 13996 ts_tablet_manager.cc:1428] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:50:21.220243 13949 raft_consensus.cc:482] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:21.220553 13999 tablet_bootstrap.cc:654] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:21.220418 13949 raft_consensus.cc:504] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.221575 13425 catalog_manager.cc:4841] TS f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337): tablet c8e13468cea94a6496e1baf9b4f56c65 (table test-table [id=d16d6c0324b549fbadf34c0defe1cb40]) successfully deleted
I20260621 02:50:21.222102 13949 leader_election.cc:288] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:21.229785 13995 ts_tablet_manager.cc:1905] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:21.231598 13999 tablet_bootstrap.cc:492] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: No bootstrap required, opened a new log
I20260621 02:50:21.231741 13999 ts_tablet_manager.cc:1397] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260621 02:50:21.232471 13999 raft_consensus.cc:348] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.232707 13999 raft_consensus.cc:374] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:21.232810 13999 raft_consensus.cc:729] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Initialized, Role: FOLLOWER
I20260621 02:50:21.233075 13999 consensus_queue.cc:260] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.235569 13999 ts_tablet_manager.cc:1428] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:50:21.237010 13995 ts_tablet_manager.cc:1918] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:21.237151 13995 log.cc:1196] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/c8e13468cea94a6496e1baf9b4f56c65
I20260621 02:50:21.238171 13995 ts_tablet_manager.cc:1939] T c8e13468cea94a6496e1baf9b4f56c65 P f5267367f49444caa6c0881a96ef3c50: Deleting consensus metadata
I20260621 02:50:21.239378 14000 tablet_bootstrap.cc:492] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: No bootstrap required, opened a new log
I20260621 02:50:21.239523 14000 ts_tablet_manager.cc:1397] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.009s
I20260621 02:50:21.239818 13425 catalog_manager.cc:4841] TS f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969): tablet c8e13468cea94a6496e1baf9b4f56c65 (table test-table [id=d16d6c0324b549fbadf34c0defe1cb40]) successfully deleted
I20260621 02:50:21.240312 14000 raft_consensus.cc:348] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.240566 14000 raft_consensus.cc:374] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:21.240660 14000 raft_consensus.cc:729] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f731a284dd5a4959bbf9f3d2b90d7faf, State: Initialized, Role: FOLLOWER
I20260621 02:50:21.240918 14000 consensus_queue.cc:260] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.241647 14000 ts_tablet_manager.cc:1428] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:21.247756 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.224127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:58597 (local address 127.11.109.3:42337)
0621 02:50:21.224324 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.224330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.226199 (+ 1869us) server_negotiation.cc:407] Connection header received
0621 02:50:21.240208 (+ 14009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.240213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.240287 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.240355 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:21.241295 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.242407 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.245130 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.246472 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.246662 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.246733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.247299 (+ 566us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.247440 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:50:21.247831 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.222826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:55989 (local address 127.11.109.2:40815)
0621 02:50:21.222992 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.222998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.223768 (+ 770us) server_negotiation.cc:407] Connection header received
0621 02:50:21.224647 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.224653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.224724 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.224781 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:21.226840 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.227961 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.245161 (+ 17200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.246556 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.246711 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.246781 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.247403 (+ 622us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.247551 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:50:21.248562 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.223811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:42337 (local address 127.11.109.1:58597)
0621 02:50:21.226104 (+ 2293us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.226121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.226692 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.240588 (+ 13896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.240593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.240631 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:21.241065 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.241078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.242931 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.242936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.244953 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.244969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.246858 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.246864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.247089 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.247136 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.247980 (+ 844us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.248118 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2178,"mutex_wait_us":26,"thread_start_us":2056,"threads_started":1}
I20260621 02:50:21.248607 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.222511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:40815 (local address 127.11.109.1:55989)
0621 02:50:21.223670 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.223690 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.224333 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.224986 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.224991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.225023 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:21.225438 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.225451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.228167 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.228172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.230375 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.230390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.246988 (+ 16598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.246996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.247216 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.247252 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.248028 (+ 776us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.248175 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":3,"thread_start_us":962,"threads_started":1}
I20260621 02:50:21.249119 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6000394e714641128fab0f920e60fb12" candidate_uuid: "f5267367f49444caa6c0881a96ef3c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" is_pre_election: true
I20260621 02:50:21.249310 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6000394e714641128fab0f920e60fb12" candidate_uuid: "f5267367f49444caa6c0881a96ef3c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" is_pre_election: true
I20260621 02:50:21.249410 13879 raft_consensus.cc:2454] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5267367f49444caa6c0881a96ef3c50 in term 0.
I20260621 02:50:21.249554 13740 raft_consensus.cc:2454] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5267367f49444caa6c0881a96ef3c50 in term 0.
I20260621 02:50:21.249936 13528 leader_election.cc:302] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f5267367f49444caa6c0881a96ef3c50, f731a284dd5a4959bbf9f3d2b90d7faf; no voters:
I20260621 02:50:21.250360 13949 raft_consensus.cc:2781] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:21.250483 13949 raft_consensus.cc:482] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:21.250574 13949 raft_consensus.cc:3037] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:21.253537 13949 raft_consensus.cc:504] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.254154 13949 leader_election.cc:288] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [CANDIDATE]: Term 1 election: Requested vote from peers 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:21.254622 13740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6000394e714641128fab0f920e60fb12" candidate_uuid: "f5267367f49444caa6c0881a96ef3c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573fba2d72cc452ebc17b628e4f6ac5d"
I20260621 02:50:21.254659 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6000394e714641128fab0f920e60fb12" candidate_uuid: "f5267367f49444caa6c0881a96ef3c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
I20260621 02:50:21.254808 13740 raft_consensus.cc:3037] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:21.254876 13879 raft_consensus.cc:3037] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:21.257865 13740 raft_consensus.cc:2454] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5267367f49444caa6c0881a96ef3c50 in term 1.
I20260621 02:50:21.258213 13879 raft_consensus.cc:2454] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5267367f49444caa6c0881a96ef3c50 in term 1.
I20260621 02:50:21.258275 13528 leader_election.cc:302] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f5267367f49444caa6c0881a96ef3c50; no voters:
I20260621 02:50:21.258555 13949 raft_consensus.cc:2781] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:21.261456 13949 raft_consensus.cc:686] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 1 LEADER]: Becoming Leader. State: Replica: f5267367f49444caa6c0881a96ef3c50, State: Running, Role: LEADER
I20260621 02:50:21.261883 13949 consensus_queue.cc:237] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:21.267555 13438 catalog_manager.cc:5495] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5267367f49444caa6c0881a96ef3c50 (127.11.109.1). New cstate: current_term: 1 leader_uuid: "f5267367f49444caa6c0881a96ef3c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } health_report { overall_health: UNKNOWN } } }
I20260621 02:50:21.307766 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.300856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.11.109.1:41969)
0621 02:50:21.301014 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.301020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.301040 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:50:21.301314 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.301322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.301417 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.301578 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.301970 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.303069 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.305109 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.305794 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.305936 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.306421 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.307102 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.307250 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.307382 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.307471 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:50:21.326700 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.319349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:58737 (local address 127.11.109.3:42337)
0621 02:50:21.319536 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.319543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.320173 (+ 630us) server_negotiation.cc:407] Connection header received
0621 02:50:21.320369 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.320375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.320487 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.320587 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.321216 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.322376 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.324467 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.325148 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.325316 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.325821 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.326002 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.326100 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.326301 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.326382 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:50:21.326788 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.319481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:42337 (local address 127.11.109.1:58737)
0621 02:50:21.320056 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.320075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.320220 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.320620 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.320625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.320639 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:21.321010 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.321021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.322540 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.322546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.324291 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.324309 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.325358 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.325365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.325540 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.325610 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.326139 (+ 529us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.326152 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.326253 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.326400 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260621 02:50:21.339780 13740 raft_consensus.cc:1261] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Refusing update from remote peer f5267367f49444caa6c0881a96ef3c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:21.340396 13879 raft_consensus.cc:1261] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Refusing update from remote peer f5267367f49444caa6c0881a96ef3c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:21.340531 13949 consensus_queue.cc:1035] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:21.341430 13949 consensus_queue.cc:1035] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:21.382476 13974 mvcc.cc:205] Tried to move back new op lower bound from 7299113866601848832 to 7299113866293067776. Current Snapshot: MvccSnapshot[applied={T|T < 7299113866601848832}]
I20260621 02:50:21.438032 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.431743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41969 (local address 127.11.109.3:51967)
0621 02:50:21.432014 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:21.432039 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:21.432147 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:21.432603 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:21.432608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:21.432621 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:50:21.432806 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.432816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.434167 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.434172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.435705 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:21.435719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.436789 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.436795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:21.436965 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.437031 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.437491 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.437502 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:50:21.437573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 02:50:21.437703 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260621 02:50:21.438644 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:21.432028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:51967 (local address 127.11.109.1:41969)
0621 02:50:21.432180 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:21.432186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:21.432230 (+ 44us) server_negotiation.cc:407] Connection header received
0621 02:50:21.432291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:21.432296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:21.432374 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:21.432453 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:21.432970 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.434008 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.435873 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:21.436578 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:21.436688 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:21.437239 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:21.437361 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:21.437442 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:21.438279 (+ 837us) server_negotiation.cc:299] Negotiation successful
0621 02:50:21.438354 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260621 02:50:21.439566 13980 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 02:50:23.036929 13439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:23.037179 13439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:23.040715 13439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Sending DeleteTablet for 3 replicas of tablet 6000394e714641128fab0f920e60fb12
I20260621 02:50:23.041332 13576 tablet_service.cc:1514] Processing DeleteTablet for tablet 6000394e714641128fab0f920e60fb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:23 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60898
I20260621 02:50:23.041402 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet 6000394e714641128fab0f920e60fb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:23 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51204
I20260621 02:50:23.041472 13858 tablet_service.cc:1514] Processing DeleteTablet for tablet 6000394e714641128fab0f920e60fb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:23 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:35524
I20260621 02:50:23.053292 14038 tablet_replica.cc:330] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: stopping tablet replica
I20260621 02:50:23.053579 14038 raft_consensus.cc:2229] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:23.054186 14038 raft_consensus.cc:2258] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:23.055330 14038 ts_tablet_manager.cc:1905] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:23.057166 14039 tablet_replica.cc:330] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: stopping tablet replica
I20260621 02:50:23.057762 14040 tablet_replica.cc:330] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: stopping tablet replica
I20260621 02:50:23.057986 14040 raft_consensus.cc:2229] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:23.058024 14039 raft_consensus.cc:2229] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:23.058208 14040 raft_consensus.cc:2258] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:23.058382 14039 raft_consensus.cc:2258] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:23.059180 14040 ts_tablet_manager.cc:1905] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:23.059341 14039 ts_tablet_manager.cc:1905] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:23.063405 14038 ts_tablet_manager.cc:1918] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:50:23.063571 14038 log.cc:1196] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/6000394e714641128fab0f920e60fb12
I20260621 02:50:23.064042 14038 ts_tablet_manager.cc:1939] T 6000394e714641128fab0f920e60fb12 P f5267367f49444caa6c0881a96ef3c50: Deleting consensus metadata
I20260621 02:50:23.065246 13425 catalog_manager.cc:4841] TS f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969): tablet 6000394e714641128fab0f920e60fb12 (table test-table [id=4fb3cdcb105a416ebd7686ee37416265]) successfully deleted
I20260621 02:50:23.065747 14040 ts_tablet_manager.cc:1918] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:50:23.065891 14040 log.cc:1196] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/6000394e714641128fab0f920e60fb12
I20260621 02:50:23.066114 14039 ts_tablet_manager.cc:1918] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:50:23.066253 14039 log.cc:1196] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/6000394e714641128fab0f920e60fb12
I20260621 02:50:23.066339 14040 ts_tablet_manager.cc:1939] T 6000394e714641128fab0f920e60fb12 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting consensus metadata
I20260621 02:50:23.066658 14039 ts_tablet_manager.cc:1939] T 6000394e714641128fab0f920e60fb12 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting consensus metadata
I20260621 02:50:23.067432 13425 catalog_manager.cc:4841] TS f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337): tablet 6000394e714641128fab0f920e60fb12 (table test-table [id=4fb3cdcb105a416ebd7686ee37416265]) successfully deleted
I20260621 02:50:23.067689 13425 catalog_manager.cc:4841] TS 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815): tablet 6000394e714641128fab0f920e60fb12 (table test-table [id=4fb3cdcb105a416ebd7686ee37416265]) successfully deleted
Jun 21 02:50:23 dist-test-slave-bc52 krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:50:23 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:23.097507 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:23.083506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.11.109.62:45907)
0621 02:50:23.084120 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:23.084124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:23.084154 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:50:23.084289 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:23.084292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:23.084352 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:23.084438 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:23.085996 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:23.086721 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:23.087839 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:23.088229 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:23.088318 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:23.092159 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:23.092164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:23.092168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:23.092199 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:23.095022 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:23.095643 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:23.095647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:23.095649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:23.095758 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:23.096168 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:23.096171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:23.096174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:23.096480 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:23.096698 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:23.096934 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 02:50:23.097085 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
I20260621 02:50:23.100798 13439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:23.101390 13439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:23.107532 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 85f6247150de4db883b3ab012af48cee (DEFAULT_TABLE table=test-table [id=ad3af69c49c64506ad99e668dbca3e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:23.107594 13715 tablet_service.cc:1467] Processing CreateTablet for tablet 85f6247150de4db883b3ab012af48cee (DEFAULT_TABLE table=test-table [id=ad3af69c49c64506ad99e668dbca3e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:23.107769 13858 tablet_service.cc:1467] Processing CreateTablet for tablet 85f6247150de4db883b3ab012af48cee (DEFAULT_TABLE table=test-table [id=ad3af69c49c64506ad99e668dbca3e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:23.107949 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f6247150de4db883b3ab012af48cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:23.108006 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f6247150de4db883b3ab012af48cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:23.108167 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f6247150de4db883b3ab012af48cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:23.113155 14049 tablet_bootstrap.cc:492] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Bootstrap starting.
I20260621 02:50:23.114686 14050 tablet_bootstrap.cc:492] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Bootstrap starting.
I20260621 02:50:23.115193 14049 tablet_bootstrap.cc:654] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:23.116394 14050 tablet_bootstrap.cc:654] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:23.117233 14051 tablet_bootstrap.cc:492] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Bootstrap starting.
I20260621 02:50:23.117705 14049 tablet_bootstrap.cc:492] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: No bootstrap required, opened a new log
I20260621 02:50:23.117867 14049 ts_tablet_manager.cc:1397] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 02:50:23.118512 14049 raft_consensus.cc:348] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.118737 14049 raft_consensus.cc:374] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:23.118764 14051 tablet_bootstrap.cc:654] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:23.118848 14049 raft_consensus.cc:729] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5267367f49444caa6c0881a96ef3c50, State: Initialized, Role: FOLLOWER
I20260621 02:50:23.119099 14049 consensus_queue.cc:260] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.119594 14049 ts_tablet_manager.cc:1428] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:50:23.121336 14050 tablet_bootstrap.cc:492] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: No bootstrap required, opened a new log
I20260621 02:50:23.121497 14050 ts_tablet_manager.cc:1397] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260621 02:50:23.121825 14051 tablet_bootstrap.cc:492] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: No bootstrap required, opened a new log
I20260621 02:50:23.121970 14051 ts_tablet_manager.cc:1397] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:50:23.122306 14050 raft_consensus.cc:348] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.122547 14050 raft_consensus.cc:374] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:23.122635 14050 raft_consensus.cc:729] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Initialized, Role: FOLLOWER
I20260621 02:50:23.122720 14051 raft_consensus.cc:348] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.122855 14050 consensus_queue.cc:260] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.123027 14051 raft_consensus.cc:374] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:23.123119 14051 raft_consensus.cc:729] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f731a284dd5a4959bbf9f3d2b90d7faf, State: Initialized, Role: FOLLOWER
I20260621 02:50:23.123342 14051 consensus_queue.cc:260] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.123879 14051 ts_tablet_manager.cc:1428] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:23.124491 14050 ts_tablet_manager.cc:1428] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:23.199385 14055 raft_consensus.cc:482] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:23.199589 14055 raft_consensus.cc:504] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.200186 14055 leader_election.cc:288] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:23.200776 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f6247150de4db883b3ab012af48cee" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50" is_pre_election: true
I20260621 02:50:23.200863 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f6247150de4db883b3ab012af48cee" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" is_pre_election: true
I20260621 02:50:23.201015 13601 raft_consensus.cc:2454] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 0.
I20260621 02:50:23.201205 13879 raft_consensus.cc:2454] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 0.
I20260621 02:50:23.201541 13668 leader_election.cc:302] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f5267367f49444caa6c0881a96ef3c50; no voters:
I20260621 02:50:23.201783 14055 raft_consensus.cc:2781] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:23.201903 14055 raft_consensus.cc:482] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:23.202003 14055 raft_consensus.cc:3037] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:23.204070 14055 raft_consensus.cc:504] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.204555 14055 leader_election.cc:288] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 election: Requested vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:23.204998 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f6247150de4db883b3ab012af48cee" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50"
I20260621 02:50:23.204998 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f6247150de4db883b3ab012af48cee" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
I20260621 02:50:23.205245 13879 raft_consensus.cc:3037] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:23.205232 13601 raft_consensus.cc:3037] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:23.207094 13879 raft_consensus.cc:2454] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 1.
I20260621 02:50:23.207286 13601 raft_consensus.cc:2454] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 1.
I20260621 02:50:23.207535 13668 leader_election.cc:302] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f731a284dd5a4959bbf9f3d2b90d7faf; no voters:
I20260621 02:50:23.207808 14055 raft_consensus.cc:2781] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:23.208204 14055 raft_consensus.cc:686] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 LEADER]: Becoming Leader. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Running, Role: LEADER
I20260621 02:50:23.208506 14055 consensus_queue.cc:237] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.211549 13438 catalog_manager.cc:5495] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2). New cstate: current_term: 1 leader_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } health_report { overall_health: UNKNOWN } } }
I20260621 02:50:23.230437 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:23.225875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.11.109.2:40815)
0621 02:50:23.226419 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:23.226423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:23.226440 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:50:23.226485 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:23.226488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:23.226543 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:23.226622 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:23.226932 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:23.227721 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:23.228901 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:23.229318 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:23.229408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:23.229741 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:23.229835 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:23.229900 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:23.230042 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 02:50:23.230120 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":334,"threads_started":1}
I20260621 02:50:23.237965 13879 raft_consensus.cc:1261] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Refusing update from remote peer 573fba2d72cc452ebc17b628e4f6ac5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:23.237957 13601 raft_consensus.cc:1261] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Refusing update from remote peer 573fba2d72cc452ebc17b628e4f6ac5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:23.238451 14055 consensus_queue.cc:1035] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:23.238698 14056 consensus_queue.cc:1035] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:23.281889 13438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:23.282083 13438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:23.284565 13438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Sending DeleteTablet for 3 replicas of tablet 85f6247150de4db883b3ab012af48cee
I20260621 02:50:23.285329 13576 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f6247150de4db883b3ab012af48cee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:23 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60898
I20260621 02:50:23.285427 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f6247150de4db883b3ab012af48cee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:23 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51204
I20260621 02:50:23.285588 14038 tablet_replica.cc:330] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: stopping tablet replica
I20260621 02:50:23.285772 14038 raft_consensus.cc:2229] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:23.285840 13858 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f6247150de4db883b3ab012af48cee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:23 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:35524
I20260621 02:50:23.285943 14039 tablet_replica.cc:330] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: stopping tablet replica
I20260621 02:50:23.285948 14038 raft_consensus.cc:2258] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:23.286069 14040 tablet_replica.cc:330] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: stopping tablet replica
I20260621 02:50:23.286130 14039 raft_consensus.cc:2229] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:23.286252 14040 raft_consensus.cc:2229] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:23.286412 14039 raft_consensus.cc:2258] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:23.286422 14040 raft_consensus.cc:2258] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:23.286332 13438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:23.286958 13438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:23.287432 14040 ts_tablet_manager.cc:1905] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:23.287803 14039 ts_tablet_manager.cc:1905] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:23.289219 14038 ts_tablet_manager.cc:1905] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:23.292558 14040 ts_tablet_manager.cc:1918] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:23.292719 14040 log.cc:1196] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/85f6247150de4db883b3ab012af48cee
I20260621 02:50:23.293085 14040 ts_tablet_manager.cc:1939] T 85f6247150de4db883b3ab012af48cee P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting consensus metadata
I20260621 02:50:23.293963 13425 catalog_manager.cc:4841] TS f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337): tablet 85f6247150de4db883b3ab012af48cee (table test-table [id=ad3af69c49c64506ad99e668dbca3e5e]) successfully deleted
I20260621 02:50:23.294279 14039 ts_tablet_manager.cc:1918] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:23.294432 14039 log.cc:1196] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/85f6247150de4db883b3ab012af48cee
I20260621 02:50:23.296049 14038 ts_tablet_manager.cc:1918] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:50:23.296204 14038 log.cc:1196] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/85f6247150de4db883b3ab012af48cee
I20260621 02:50:23.296526 14039 ts_tablet_manager.cc:1939] T 85f6247150de4db883b3ab012af48cee P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting consensus metadata
I20260621 02:50:23.296618 14038 ts_tablet_manager.cc:1939] T 85f6247150de4db883b3ab012af48cee P f5267367f49444caa6c0881a96ef3c50: Deleting consensus metadata
I20260621 02:50:23.297466 13425 catalog_manager.cc:4841] TS 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815): tablet 85f6247150de4db883b3ab012af48cee (table test-table [id=ad3af69c49c64506ad99e668dbca3e5e]) successfully deleted
I20260621 02:50:23.297659 13425 catalog_manager.cc:4841] TS f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969): tablet 85f6247150de4db883b3ab012af48cee (table test-table [id=ad3af69c49c64506ad99e668dbca3e5e]) successfully deleted
I20260621 02:50:23.298667 13576 tablet_service.cc:1467] Processing CreateTablet for tablet af506b1f31da463aa3c6d35ed548ad65 (DEFAULT_TABLE table=test-table [id=dece074ca3674bfb912d229e931b2a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:23.298905 13715 tablet_service.cc:1467] Processing CreateTablet for tablet af506b1f31da463aa3c6d35ed548ad65 (DEFAULT_TABLE table=test-table [id=dece074ca3674bfb912d229e931b2a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:23.299073 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af506b1f31da463aa3c6d35ed548ad65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:23.299134 13858 tablet_service.cc:1467] Processing CreateTablet for tablet af506b1f31da463aa3c6d35ed548ad65 (DEFAULT_TABLE table=test-table [id=dece074ca3674bfb912d229e931b2a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:23.299278 13715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af506b1f31da463aa3c6d35ed548ad65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:23.299495 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af506b1f31da463aa3c6d35ed548ad65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:23.303264 14049 tablet_bootstrap.cc:492] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Bootstrap starting.
I20260621 02:50:23.304484 14049 tablet_bootstrap.cc:654] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:23.305238 14050 tablet_bootstrap.cc:492] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Bootstrap starting.
I20260621 02:50:23.305684 14051 tablet_bootstrap.cc:492] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Bootstrap starting.
I20260621 02:50:23.306973 14050 tablet_bootstrap.cc:654] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:23.307825 14051 tablet_bootstrap.cc:654] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:23.309048 14050 tablet_bootstrap.cc:492] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: No bootstrap required, opened a new log
I20260621 02:50:23.309248 14050 ts_tablet_manager.cc:1397] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:50:23.309516 14049 tablet_bootstrap.cc:492] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: No bootstrap required, opened a new log
I20260621 02:50:23.309700 14049 ts_tablet_manager.cc:1397] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260621 02:50:23.310001 14050 raft_consensus.cc:348] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.310209 14050 raft_consensus.cc:374] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:23.310303 14050 raft_consensus.cc:729] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Initialized, Role: FOLLOWER
I20260621 02:50:23.310411 14049 raft_consensus.cc:348] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.310685 14049 raft_consensus.cc:374] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:23.310580 14050 consensus_queue.cc:260] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.310791 14049 raft_consensus.cc:729] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5267367f49444caa6c0881a96ef3c50, State: Initialized, Role: FOLLOWER
I20260621 02:50:23.311084 14050 ts_tablet_manager.cc:1428] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:50:23.311012 14049 consensus_queue.cc:260] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.311491 14049 ts_tablet_manager.cc:1428] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:50:23.313138 14051 tablet_bootstrap.cc:492] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: No bootstrap required, opened a new log
I20260621 02:50:23.313351 14051 ts_tablet_manager.cc:1397] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260621 02:50:23.313926 14051 raft_consensus.cc:348] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.314168 14051 raft_consensus.cc:374] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:23.314262 14051 raft_consensus.cc:729] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f731a284dd5a4959bbf9f3d2b90d7faf, State: Initialized, Role: FOLLOWER
I20260621 02:50:23.314502 14051 consensus_queue.cc:260] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.315130 14051 ts_tablet_manager.cc:1428] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:23.540421 14063 raft_consensus.cc:482] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:23.540652 14063 raft_consensus.cc:504] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.541385 14063 leader_election.cc:288] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:23.542011 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af506b1f31da463aa3c6d35ed548ad65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" is_pre_election: true
I20260621 02:50:23.542022 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af506b1f31da463aa3c6d35ed548ad65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50" is_pre_election: true
I20260621 02:50:23.542332 13879 raft_consensus.cc:2454] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 0.
I20260621 02:50:23.542344 13601 raft_consensus.cc:2454] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 0.
I20260621 02:50:23.542775 13668 leader_election.cc:302] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f731a284dd5a4959bbf9f3d2b90d7faf; no voters:
I20260621 02:50:23.543045 14063 raft_consensus.cc:2781] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:23.543180 14063 raft_consensus.cc:482] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:23.543284 14063 raft_consensus.cc:3037] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:23.545377 14063 raft_consensus.cc:504] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.545866 14063 leader_election.cc:288] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 election: Requested vote from peers f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969), f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337)
I20260621 02:50:23.546351 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af506b1f31da463aa3c6d35ed548ad65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5267367f49444caa6c0881a96ef3c50"
I20260621 02:50:23.546396 13879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af506b1f31da463aa3c6d35ed548ad65" candidate_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf"
I20260621 02:50:23.546540 13601 raft_consensus.cc:3037] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:23.546588 13879 raft_consensus.cc:3037] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:23.548451 13601 raft_consensus.cc:2454] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 1.
I20260621 02:50:23.548542 13879 raft_consensus.cc:2454] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573fba2d72cc452ebc17b628e4f6ac5d in term 1.
I20260621 02:50:23.548900 13668 leader_election.cc:302] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573fba2d72cc452ebc17b628e4f6ac5d, f5267367f49444caa6c0881a96ef3c50; no voters:
I20260621 02:50:23.549211 14063 raft_consensus.cc:2781] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:23.549388 14063 raft_consensus.cc:686] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 LEADER]: Becoming Leader. State: Replica: 573fba2d72cc452ebc17b628e4f6ac5d, State: Running, Role: LEADER
I20260621 02:50:23.549639 14063 consensus_queue.cc:237] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } }
I20260621 02:50:23.552065 13438 catalog_manager.cc:5495] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2). New cstate: current_term: 1 leader_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "573fba2d72cc452ebc17b628e4f6ac5d" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 40815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 } health_report { overall_health: UNKNOWN } } }
I20260621 02:50:23.617414 13879 raft_consensus.cc:1261] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Refusing update from remote peer 573fba2d72cc452ebc17b628e4f6ac5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:23.617409 13601 raft_consensus.cc:1261] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Refusing update from remote peer 573fba2d72cc452ebc17b628e4f6ac5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:23.617995 14055 consensus_queue.cc:1035] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f731a284dd5a4959bbf9f3d2b90d7faf" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:23.618258 14063 consensus_queue.cc:1035] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5267367f49444caa6c0881a96ef3c50" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:50:23.636009 14021 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 02:50:25.211326 13439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:25.211519 13439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:25.215199 13439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d78f6afcfedc4ac69aa600839e635147: Sending DeleteTablet for 3 replicas of tablet af506b1f31da463aa3c6d35ed548ad65
I20260621 02:50:25.215884 13576 tablet_service.cc:1514] Processing DeleteTablet for tablet af506b1f31da463aa3c6d35ed548ad65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:25 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60898
I20260621 02:50:25.215977 13858 tablet_service.cc:1514] Processing DeleteTablet for tablet af506b1f31da463aa3c6d35ed548ad65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:25 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:35524
I20260621 02:50:25.216555 14093 tablet_replica.cc:330] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: stopping tablet replica
I20260621 02:50:25.216552 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13510
I20260621 02:50:25.216784 14093 raft_consensus.cc:2229] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:25.217041 14093 raft_consensus.cc:2258] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:25.216984 13715 tablet_service.cc:1514] Processing DeleteTablet for tablet af506b1f31da463aa3c6d35ed548ad65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:25 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51204
I20260621 02:50:25.217664 14095 tablet_replica.cc:330] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: stopping tablet replica
I20260621 02:50:25.218312 14095 raft_consensus.cc:2229] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:25.218478 14093 ts_tablet_manager.cc:1905] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:25.218672 14095 raft_consensus.cc:2258] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:25.219700 14095 ts_tablet_manager.cc:1905] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:25.227499 14093 ts_tablet_manager.cc:1918] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:50:25.227558 14095 ts_tablet_manager.cc:1918] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:50:25.227742 14093 log.cc:1196] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/af506b1f31da463aa3c6d35ed548ad65
I20260621 02:50:25.227772 14095 log.cc:1196] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/af506b1f31da463aa3c6d35ed548ad65
I20260621 02:50:25.228281 14095 ts_tablet_manager.cc:1939] T af506b1f31da463aa3c6d35ed548ad65 P 573fba2d72cc452ebc17b628e4f6ac5d: Deleting consensus metadata
I20260621 02:50:25.228358 14093 ts_tablet_manager.cc:1939] T af506b1f31da463aa3c6d35ed548ad65 P f5267367f49444caa6c0881a96ef3c50: Deleting consensus metadata
I20260621 02:50:25.228482 14094 tablet_replica.cc:330] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: stopping tablet replica
I20260621 02:50:25.228750 14094 raft_consensus.cc:2229] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:25.229020 14094 raft_consensus.cc:2258] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:25.229635 13425 catalog_manager.cc:4841] TS f5267367f49444caa6c0881a96ef3c50 (127.11.109.1:41969): tablet af506b1f31da463aa3c6d35ed548ad65 (table test-table [id=dece074ca3674bfb912d229e931b2a9d]) successfully deleted
I20260621 02:50:25.230132 14094 ts_tablet_manager.cc:1905] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:25.233425 13425 catalog_manager.cc:4841] TS 573fba2d72cc452ebc17b628e4f6ac5d (127.11.109.2:40815): tablet af506b1f31da463aa3c6d35ed548ad65 (table test-table [id=dece074ca3674bfb912d229e931b2a9d]) successfully deleted
I20260621 02:50:25.234498 14094 ts_tablet_manager.cc:1918] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:50:25.234647 14094 log.cc:1196] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/af506b1f31da463aa3c6d35ed548ad65
I20260621 02:50:25.235013 14094 ts_tablet_manager.cc:1939] T af506b1f31da463aa3c6d35ed548ad65 P f731a284dd5a4959bbf9f3d2b90d7faf: Deleting consensus metadata
I20260621 02:50:25.235917 13425 catalog_manager.cc:4841] TS f731a284dd5a4959bbf9f3d2b90d7faf (127.11.109.3:42337): tablet af506b1f31da463aa3c6d35ed548ad65 (table test-table [id=dece074ca3674bfb912d229e931b2a9d]) successfully deleted
Jun 21 02:50:25 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:25.247841 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:25.218337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.11.109.1:41969)
0621 02:50:25.233850 (+ 15513us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:25.233859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:25.234352 (+ 493us) server_negotiation.cc:407] Connection header received
0621 02:50:25.234557 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:25.234568 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:25.234668 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:25.234782 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:25.236364 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:25.237106 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:25.238332 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:25.238701 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:25.238792 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:25.242869 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:25.242883 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:25.242899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:25.242927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:25.245593 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:25.246263 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:25.246270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:25.246272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:25.246391 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:25.246696 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:25.246699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:25.246701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:25.246997 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:25.247228 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:25.247429 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:50:25.247593 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15357,"thread_start_us":15257,"threads_started":1}
W20260621 02:50:25.248483 13641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53938
W20260621 02:50:25.249090 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13510: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:25.249397 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 13510
I20260621 02:50:25.274489 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13650
Jun 21 02:50:25 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:25.290198 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:25.275200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.11.109.2:40815)
0621 02:50:25.275762 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:25.275767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:25.275782 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:50:25.275834 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:25.275838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:25.275902 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:25.276039 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:25.277440 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:25.278798 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:25.280132 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:25.280622 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:25.280765 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:25.284782 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:25.284794 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:25.284808 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:25.284838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:25.287895 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:25.288500 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:25.288506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:25.288508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:25.288628 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:25.288919 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:25.288922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:25.288924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:25.289339 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:25.289560 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:25.289783 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 02:50:25.289948 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":283,"threads_started":1}
W20260621 02:50:25.290889 13780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40540
W20260621 02:50:25.291493 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:25.291636 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 13650
I20260621 02:50:25.313891 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13789
Jun 21 02:50:25 dist-test-slave-bc52 krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:25.328114 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:25.314540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.11.109.3:42337)
0621 02:50:25.315032 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:25.315037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:25.315058 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:50:25.315109 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:25.315113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:25.315168 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:25.315287 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:25.316505 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:25.317271 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:25.318460 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:25.318877 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:25.319009 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:25.322959 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:25.322971 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:25.322988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:25.323025 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:25.325930 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:25.326498 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:25.326505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:25.326508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:25.326630 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:25.326924 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:25.326927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:25.326929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:25.327254 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:25.327468 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:25.327701 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 02:50:25.327849 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
W20260621 02:50:25.328732 13919 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34820
W20260621 02:50:25.329368 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13789: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:25.329532 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 13789
I20260621 02:50:25.350351 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 13407
I20260621 02:50:25.351486 13479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:25.464607 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 13407
2026-06-21T02:50:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8379 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:25 dist-test-slave-bc52 krb5kdc[14103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:25 dist-test-slave-bc52 krb5kdc[14103](info): set up 2 sockets
Jun 21 02:50:25 dist-test-slave-bc52 krb5kdc[14103](info): commencing operation
krb5kdc: starting...
W20260621 02:50:26.553539 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:26 dist-test-slave-bc52 krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:26.735038 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:32873
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:27.045116 14116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:27.045399 14116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:27.045576 14116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:27.052443 14116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:27.052570 14116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:27.052665 14116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:27.052767 14116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:27.052850 14116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:27.062086 14116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:27.063295 14116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:27.064522 14116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:27.076570 14121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:27.076625 14122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:27.076679 14116 server_base.cc:1034] running on GCE node
W20260621 02:50:27.076777 14124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:27.077805 14116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:27.079097 14116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:27.080315 14116 hybrid_clock.cc:648] HybridClock initialized: now 1782010227080256 us; error 55 us; skew 500 ppm
I20260621 02:50:27.080641 14116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:27 dist-test-slave-bc52 krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010227, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:27.083634 14116 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:27.084985 14116 webserver.cc:460] Webserver started at http://127.11.109.62:40235/ using document root <none> and password file <none>
I20260621 02:50:27.085866 14116 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:27.086045 14116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:27.086457 14116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:27.089630 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "675550c877de48fb86403d5e2487e755"
format_stamp: "Formatted at 2026-06-21 02:50:27 on dist-test-slave-bc52"
server_key: "ab4f9140acdc633252df1e6d35cbe8ff"
server_key_iv: "6965dd473dabec328120f92a81dad20f"
server_key_version: "encryptionkey@0"
I20260621 02:50:27.090425 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "675550c877de48fb86403d5e2487e755"
format_stamp: "Formatted at 2026-06-21 02:50:27 on dist-test-slave-bc52"
server_key: "ab4f9140acdc633252df1e6d35cbe8ff"
server_key_iv: "6965dd473dabec328120f92a81dad20f"
server_key_version: "encryptionkey@0"
I20260621 02:50:27.095867 14116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260621 02:50:27.099392 14131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:27.100689 14116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:27.100875 14116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "675550c877de48fb86403d5e2487e755"
format_stamp: "Formatted at 2026-06-21 02:50:27 on dist-test-slave-bc52"
server_key: "ab4f9140acdc633252df1e6d35cbe8ff"
server_key_iv: "6965dd473dabec328120f92a81dad20f"
server_key_version: "encryptionkey@0"
I20260621 02:50:27.101083 14116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:27.123183 14116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:27.127398 14116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:27.127702 14116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:27.147024 14116 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:32873
I20260621 02:50:27.147042 14182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:32873 every 8 connection(s)
I20260621 02:50:27.148494 14116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:27.152236 14183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:27.152351 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14116
I20260621 02:50:27.152566 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:27.153352 11700 external_mini_cluster.cc:1428] Setting key 8165bb6a86f6491878f534471fe1c2d5
I20260621 02:50:27.166007 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755: Bootstrap starting.
Jun 21 02:50:27 dist-test-slave-bc52 krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:27.169505 14183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:27.170535 14183 log.cc:824] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:27.173583 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755: No bootstrap required, opened a new log
I20260621 02:50:27.177018 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:27.155461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.11.109.62:32873)
0621 02:50:27.156305 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:27.156318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:27.156362 (+ 44us) server_negotiation.cc:407] Connection header received
0621 02:50:27.158493 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:27.158517 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:27.158921 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:27.159272 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:27.161000 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:27.162355 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:27.164124 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:27.164763 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:27.164889 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:27.169293 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:27.169300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:27.169314 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:27.169352 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:27.172863 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:27.173521 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:27.173528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:27.173537 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:27.173759 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:27.174105 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:27.174110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:27.174116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:27.174741 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:27.175031 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:27.175324 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 02:50:27.175750 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":397,"threads_started":1}
I20260621 02:50:27.182412 14183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675550c877de48fb86403d5e2487e755" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32873 } }
I20260621 02:50:27.182722 14183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:27.182792 14183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675550c877de48fb86403d5e2487e755, State: Initialized, Role: FOLLOWER
I20260621 02:50:27.183291 14183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675550c877de48fb86403d5e2487e755" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32873 } }
I20260621 02:50:27.183477 14183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:27.183575 14183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:27.183701 14183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:27.185938 14183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675550c877de48fb86403d5e2487e755" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32873 } }
I20260621 02:50:27.186349 14183 leader_election.cc:302] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 675550c877de48fb86403d5e2487e755; no voters:
I20260621 02:50:27.186882 14183 leader_election.cc:288] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:27.186997 14188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:27.187659 14188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [term 1 LEADER]: Becoming Leader. State: Replica: 675550c877de48fb86403d5e2487e755, State: Running, Role: LEADER
I20260621 02:50:27.188117 14188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675550c877de48fb86403d5e2487e755" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32873 } }
I20260621 02:50:27.188894 14183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:27.190922 14190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 675550c877de48fb86403d5e2487e755. Latest consensus state: current_term: 1 leader_uuid: "675550c877de48fb86403d5e2487e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675550c877de48fb86403d5e2487e755" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32873 } } }
I20260621 02:50:27.191342 14190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:27.193051 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "675550c877de48fb86403d5e2487e755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675550c877de48fb86403d5e2487e755" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32873 } } }
I20260621 02:50:27.193468 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:27.193733 14195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:27.198490 14195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:27.207227 14195 catalog_manager.cc:1347] Generated new cluster ID: a085e77a38e741abb04fd4e159d03b53
I20260621 02:50:27.207350 14195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:27.221390 14195 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:27.222628 14195 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:27.232012 14195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 675550c877de48fb86403d5e2487e755: Generated new TSK 0
I20260621 02:50:27.232786 14195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:27.284832 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32873
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:27.614956 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:27.615214 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:27.615379 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:27.622499 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:27.622650 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:27.622834 14209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:27.632359 14209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:27.633692 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:27.635025 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:27.647691 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:27.647686 14217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:27.648079 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:27.648465 14209 server_base.cc:1034] running on GCE node
I20260621 02:50:27.649039 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:27.649928 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:27.651151 14209 hybrid_clock.cc:648] HybridClock initialized: now 1782010227651114 us; error 51 us; skew 500 ppm
I20260621 02:50:27.651470 14209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:27 dist-test-slave-bc52 krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010227, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:27.654477 14209 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:27.655758 14209 webserver.cc:460] Webserver started at http://127.11.109.1:36593/ using document root <none> and password file <none>
I20260621 02:50:27.656352 14209 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:27.656445 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:27.656673 14209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:27.659389 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "8c728851f453466c981cb2536b97d51d"
format_stamp: "Formatted at 2026-06-21 02:50:27 on dist-test-slave-bc52"
server_key: "7b43ea1202fbc560fb7bdb26215af3dd"
server_key_iv: "e363e5898d8021e3583d844d6a4cafc5"
server_key_version: "encryptionkey@0"
I20260621 02:50:27.660065 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "8c728851f453466c981cb2536b97d51d"
format_stamp: "Formatted at 2026-06-21 02:50:27 on dist-test-slave-bc52"
server_key: "7b43ea1202fbc560fb7bdb26215af3dd"
server_key_iv: "e363e5898d8021e3583d844d6a4cafc5"
server_key_version: "encryptionkey@0"
I20260621 02:50:27.664855 14209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260621 02:50:27.667999 14224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:27.669312 14209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:27.669468 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "8c728851f453466c981cb2536b97d51d"
format_stamp: "Formatted at 2026-06-21 02:50:27 on dist-test-slave-bc52"
server_key: "7b43ea1202fbc560fb7bdb26215af3dd"
server_key_iv: "e363e5898d8021e3583d844d6a4cafc5"
server_key_version: "encryptionkey@0"
I20260621 02:50:27.669628 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:27.692175 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:27.696419 14209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:27.696645 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:27.697517 14209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:27.698686 14209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:27.698768 14209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:27.698846 14209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:27.698889 14209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:27.730652 14209 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:35825
I20260621 02:50:27.730739 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:35825 every 8 connection(s)
I20260621 02:50:27.732046 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:27.733918 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14209
I20260621 02:50:27.734145 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:27.734846 11700 external_mini_cluster.cc:1428] Setting key 5169c03828d1ef4ad151f10c0b70d9f7
Jun 21 02:50:27 dist-test-slave-bc52 krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010227, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:27.755183 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:27.734699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:41723 (local address 127.11.109.62:32873)
0621 02:50:27.735134 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:27.735140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:27.736185 (+ 1045us) server_negotiation.cc:407] Connection header received
0621 02:50:27.738804 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:27.738808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:27.738874 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:27.738961 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:27.741098 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:27.742282 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:27.745021 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:27.745767 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:27.745896 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:27.749630 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:27.749634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:27.749638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:27.749668 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:27.752214 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:27.752917 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:27.752923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:27.752927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:27.753043 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:27.753492 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:27.753499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:27.753502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:27.753791 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:27.753951 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:27.754617 (+ 666us) server_negotiation.cc:299] Negotiation successful
0621 02:50:27.754838 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260621 02:50:27.756170 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:27.735245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:32873 (local address 127.11.109.1:41723)
0621 02:50:27.736042 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:27.736078 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:27.738473 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:27.739261 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:27.739275 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:27.740050 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:27.740885 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:27.740918 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:27.742473 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:27.742481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:27.744865 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:27.744878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:27.746034 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:27.746043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:27.746273 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:27.747027 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:27.747081 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:27.749414 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:27.752348 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:27.752356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:27.752372 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:27.752797 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:27.753159 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:27.753164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:27.753168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:27.753385 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:27.753921 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:27.753929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:27.754256 (+ 327us) client_negotiation.cc:769] Sending connection context
0621 02:50:27.754508 (+ 252us) client_negotiation.cc:240] Negotiation successful
0621 02:50:27.754991 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":333,"threads_started":1}
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:50:27.757704 14337 heartbeater.cc:347] Connected to a master server at 127.11.109.62:32873
I20260621 02:50:27.758036 14337 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:27.759061 14337 heartbeater.cc:511] Master 127.11.109.62:32873 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:50:27.761556 14148 ts_manager.cc:194] Registered new tserver with Master: 8c728851f453466c981cb2536b97d51d (127.11.109.1:35825)
I20260621 02:50:27.763703 14148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:41723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:27.786852 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32873
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:28.112062 14343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:28.112329 14343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:28.112488 14343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:28.119735 14343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:28.119863 14343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:28.120041 14343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:28.130031 14343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:28.131285 14343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:28.132705 14343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:28.145246 14349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:28.145267 14348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:28.145324 14351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:28.146239 14343 server_base.cc:1034] running on GCE node
I20260621 02:50:28.146929 14343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:28.148063 14343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:28.149277 14343 hybrid_clock.cc:648] HybridClock initialized: now 1782010228149224 us; error 63 us; skew 500 ppm
I20260621 02:50:28.149622 14343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:28 dist-test-slave-bc52 krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:28.153276 14343 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:28.158241 14343 webserver.cc:460] Webserver started at http://127.11.109.2:46437/ using document root <none> and password file <none>
I20260621 02:50:28.158902 14343 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:28.159044 14343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:28.159350 14343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:28.162206 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "c6c836fe0a3e4b40b199e7fbbda1739e"
format_stamp: "Formatted at 2026-06-21 02:50:28 on dist-test-slave-bc52"
server_key: "c9aedc89519188100c84c7721c5a74ac"
server_key_iv: "e4fb4874116dc10b45f6e079b97899a6"
server_key_version: "encryptionkey@0"
I20260621 02:50:28.162983 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "c6c836fe0a3e4b40b199e7fbbda1739e"
format_stamp: "Formatted at 2026-06-21 02:50:28 on dist-test-slave-bc52"
server_key: "c9aedc89519188100c84c7721c5a74ac"
server_key_iv: "e4fb4874116dc10b45f6e079b97899a6"
server_key_version: "encryptionkey@0"
I20260621 02:50:28.168017 14343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:50:28.171314 14358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:28.172535 14343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:28.172703 14343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "c6c836fe0a3e4b40b199e7fbbda1739e"
format_stamp: "Formatted at 2026-06-21 02:50:28 on dist-test-slave-bc52"
server_key: "c9aedc89519188100c84c7721c5a74ac"
server_key_iv: "e4fb4874116dc10b45f6e079b97899a6"
server_key_version: "encryptionkey@0"
I20260621 02:50:28.172981 14343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:28.190495 14343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:28.194353 14343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:28.194654 14343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:28.195518 14343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:28.196746 14343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:28.196817 14343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:28.196899 14343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:28.196943 14343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:28.229444 14343 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:37307
I20260621 02:50:28.229488 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:37307 every 8 connection(s)
I20260621 02:50:28.230708 14343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:28.238869 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14343
I20260621 02:50:28.239068 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:28.239481 11700 external_mini_cluster.cc:1428] Setting key e384f6a37bbba23a26aeed5836705e86
Jun 21 02:50:28 dist-test-slave-bc52 krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:28.250967 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:28.233237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:35409 (local address 127.11.109.62:32873)
0621 02:50:28.233431 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:28.233438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:28.234702 (+ 1264us) server_negotiation.cc:407] Connection header received
0621 02:50:28.236408 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:28.236413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:28.236482 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:28.236570 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:28.238076 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.238790 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.240580 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.241086 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.241249 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:28.244794 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:28.244803 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:28.244811 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:28.244857 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:28.247863 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:28.248785 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:28.248789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:28.248791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:28.248907 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:28.249401 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:28.249405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:28.249407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:28.249646 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:28.249754 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:28.250566 (+ 812us) server_negotiation.cc:299] Negotiation successful
0621 02:50:28.250724 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:50:28.252511 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:28.233865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:32873 (local address 127.11.109.2:35409)
0621 02:50:28.234533 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:28.234568 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:28.236167 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:28.236796 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:28.236803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:28.237379 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:28.237930 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:28.237946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.238950 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.238955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:28.240427 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:28.240438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.241338 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.241346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:28.241592 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:28.242285 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:28.242312 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:28.244546 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:28.248001 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:28.248012 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:28.248031 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:28.248667 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:28.249013 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:28.249021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:28.249028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:28.249290 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:28.249776 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:28.249786 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:28.250210 (+ 424us) client_negotiation.cc:769] Sending connection context
0621 02:50:28.250496 (+ 286us) client_negotiation.cc:240] Negotiation successful
0621 02:50:28.251047 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":272,"threads_started":1}
I20260621 02:50:28.253978 14471 heartbeater.cc:347] Connected to a master server at 127.11.109.62:32873
I20260621 02:50:28.254287 14471 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:28.254971 14471 heartbeater.cc:511] Master 127.11.109.62:32873 requested a full tablet report, sending...
I20260621 02:50:28.256353 14148 ts_manager.cc:194] Registered new tserver with Master: c6c836fe0a3e4b40b199e7fbbda1739e (127.11.109.2:37307)
I20260621 02:50:28.257324 14148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:35409
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:28.282459 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32873
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:28.591795 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:28.592036 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:28.592208 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:28.599264 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:28.599351 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:28.599478 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:28.608392 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:28.609593 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:28.610821 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:28.623543 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:28.623519 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:28.623765 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:28.624293 14476 server_base.cc:1034] running on GCE node
I20260621 02:50:28.624794 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:28.625654 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:28.626849 14476 hybrid_clock.cc:648] HybridClock initialized: now 1782010228626808 us; error 40 us; skew 500 ppm
I20260621 02:50:28.627136 14476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:28 dist-test-slave-bc52 krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:28.630208 14476 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:28.631609 14476 webserver.cc:460] Webserver started at http://127.11.109.3:34857/ using document root <none> and password file <none>
I20260621 02:50:28.632246 14476 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:28.632373 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:28.632687 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:28.635521 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "eddba92676a948df98800e2dd10d057d"
format_stamp: "Formatted at 2026-06-21 02:50:28 on dist-test-slave-bc52"
server_key: "3e36cf5b3df80ad568c925d67ccd4108"
server_key_iv: "57f03c95c4142a507063e2f3e3fdee65"
server_key_version: "encryptionkey@0"
I20260621 02:50:28.636279 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "eddba92676a948df98800e2dd10d057d"
format_stamp: "Formatted at 2026-06-21 02:50:28 on dist-test-slave-bc52"
server_key: "3e36cf5b3df80ad568c925d67ccd4108"
server_key_iv: "57f03c95c4142a507063e2f3e3fdee65"
server_key_version: "encryptionkey@0"
I20260621 02:50:28.641095 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260621 02:50:28.644407 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:28.645651 14476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:50:28.645834 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "eddba92676a948df98800e2dd10d057d"
format_stamp: "Formatted at 2026-06-21 02:50:28 on dist-test-slave-bc52"
server_key: "3e36cf5b3df80ad568c925d67ccd4108"
server_key_iv: "57f03c95c4142a507063e2f3e3fdee65"
server_key_version: "encryptionkey@0"
I20260621 02:50:28.646025 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:28.659755 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:28.663404 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:28.663688 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:28.664542 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:28.665880 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:28.665988 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:28.666136 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:28.666206 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:28.698375 14476 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:43121
I20260621 02:50:28.698460 14603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:43121 every 8 connection(s)
I20260621 02:50:28.699651 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:28.708638 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14476
I20260621 02:50:28.708827 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:28.709363 11700 external_mini_cluster.cc:1428] Setting key 141ce57117d220ff42e30ffc56e76b22
Jun 21 02:50:28 dist-test-slave-bc52 krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010228, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:28.719238 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:28.702119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:52613 (local address 127.11.109.62:32873)
0621 02:50:28.702287 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:28.702292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:28.703356 (+ 1064us) server_negotiation.cc:407] Connection header received
0621 02:50:28.705208 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:28.705213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:28.705320 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:28.705431 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:28.706787 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.707556 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.709567 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.710094 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.710249 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:28.713923 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:28.713931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:28.713941 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:28.713971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:28.716353 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:28.717036 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:28.717044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:28.717047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:28.717234 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:28.717659 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:28.717666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:28.717669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:28.717927 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:28.718079 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:28.718690 (+ 611us) server_negotiation.cc:299] Negotiation successful
0621 02:50:28.718892 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:50:28.720137 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:28.702516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:32873 (local address 127.11.109.3:52613)
0621 02:50:28.703198 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:28.703237 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:28.704928 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:28.705651 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:28.705659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:28.706123 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:28.706627 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:28.706642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.707713 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.707717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:28.709419 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:28.709432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:28.710324 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:28.710334 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:28.710583 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:28.711402 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:28.711434 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:28.713734 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:28.716490 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:28.716498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:28.716511 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:28.716911 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:28.717355 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:28.717360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:28.717363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:28.717539 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:28.718052 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:28.718059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:28.718356 (+ 297us) client_negotiation.cc:769] Sending connection context
0621 02:50:28.718616 (+ 260us) client_negotiation.cc:240] Negotiation successful
0621 02:50:28.719066 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":264,"threads_started":1}
I20260621 02:50:28.721494 14604 heartbeater.cc:347] Connected to a master server at 127.11.109.62:32873
I20260621 02:50:28.721742 14604 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:28.722337 14604 heartbeater.cc:511] Master 127.11.109.62:32873 requested a full tablet report, sending...
I20260621 02:50:28.723740 14148 ts_manager.cc:194] Registered new tserver with Master: eddba92676a948df98800e2dd10d057d (127.11.109.3:43121)
I20260621 02:50:28.724514 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:50:28.724778 14148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:52613
W20260621 02:50:28.753569 14339 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:28.750969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.11.109.62:32873)
0621 02:50:28.751290 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:28.751299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:28.751666 (+ 367us) server_negotiation.cc:407] Connection header received
0621 02:50:28.751876 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:28.751884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:28.751965 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:28.752072 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:28.753313 (+ 1241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45540: BlockingRecv error: recv got EOF from 127.0.0.1:45540 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260621 02:50:28.756716 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14209
W20260621 02:50:28.759298 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14209: Not authorized: Client connection negotiation failed: client connection to 127.11.109.1:35825: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:50:28.759498 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14209
I20260621 02:50:28.775028 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14343
W20260621 02:50:28.777676 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14343: Not authorized: Client connection negotiation failed: client connection to 127.11.109.2:37307: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:50:28.777837 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14343
I20260621 02:50:28.792563 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14476
W20260621 02:50:28.795347 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14476: Not authorized: Client connection negotiation failed: client connection to 127.11.109.3:43121: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:50:28.795461 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14476
I20260621 02:50:28.810137 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14116
I20260621 02:50:28.811210 14178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:28.911434 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14116
2026-06-21T02:50:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3444 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:29 dist-test-slave-bc52 krb5kdc[14621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:29 dist-test-slave-bc52 krb5kdc[14621](info): set up 2 sockets
Jun 21 02:50:29 dist-test-slave-bc52 krb5kdc[14621](info): commencing operation
krb5kdc: starting...
W20260621 02:50:29.985517 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:30 dist-test-slave-bc52 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:30.167758 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:42717
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:42717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:30.480669 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:30.480937 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:30.481086 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:30.488101 14634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:30.488198 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:30.488278 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:30.488381 14634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:30.488466 14634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:30.498052 14634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:42717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:42717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:30.499279 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:30.500538 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:30.511688 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:30.511723 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:30.511951 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:30.512670 14634 server_base.cc:1034] running on GCE node
I20260621 02:50:30.513389 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:30.514639 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:30.515836 14634 hybrid_clock.cc:648] HybridClock initialized: now 1782010230515805 us; error 27 us; skew 500 ppm
I20260621 02:50:30.516114 14634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:30 dist-test-slave-bc52 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010230, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:30.519137 14634 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:30.520452 14634 webserver.cc:460] Webserver started at http://127.11.109.62:34813/ using document root <none> and password file <none>
I20260621 02:50:30.521140 14634 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:30.521329 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:30.521651 14634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:30.524477 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "04bfb1eb517c46068d54040cf2777fe1"
format_stamp: "Formatted at 2026-06-21 02:50:30 on dist-test-slave-bc52"
server_key: "a6adea3448d290029153d33e5f220c09"
server_key_iv: "aa42101a6c0b6288e88b99bb26c45b0e"
server_key_version: "encryptionkey@0"
I20260621 02:50:30.525303 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "04bfb1eb517c46068d54040cf2777fe1"
format_stamp: "Formatted at 2026-06-21 02:50:30 on dist-test-slave-bc52"
server_key: "a6adea3448d290029153d33e5f220c09"
server_key_iv: "aa42101a6c0b6288e88b99bb26c45b0e"
server_key_version: "encryptionkey@0"
I20260621 02:50:30.530179 14634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:50:30.533376 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:30.534667 14634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:50:30.534864 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "04bfb1eb517c46068d54040cf2777fe1"
format_stamp: "Formatted at 2026-06-21 02:50:30 on dist-test-slave-bc52"
server_key: "a6adea3448d290029153d33e5f220c09"
server_key_iv: "aa42101a6c0b6288e88b99bb26c45b0e"
server_key_version: "encryptionkey@0"
I20260621 02:50:30.535068 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:30.553706 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:30.558027 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:30.558312 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:30.576000 14634 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:42717
I20260621 02:50:30.576020 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:42717 every 8 connection(s)
I20260621 02:50:30.577370 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:30.580946 14701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:30.583598 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14634
I20260621 02:50:30.583828 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:30.584592 11700 external_mini_cluster.cc:1428] Setting key 8c87c01e62f8ba28bb79f91475082623
I20260621 02:50:30.593509 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1: Bootstrap starting.
I20260621 02:50:30.597244 14701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:30.598625 14701 log.cc:824] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:50:30 dist-test-slave-bc52 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:30.601374 14701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1: No bootstrap required, opened a new log
I20260621 02:50:30.607488 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:30.586488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.11.109.62:42717)
0621 02:50:30.587358 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:30.587370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:30.587410 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:50:30.589021 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:30.589044 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:30.589516 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:30.589868 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:30.591682 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:30.592928 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:30.594646 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:30.595355 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:30.595483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:30.600339 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:30.600347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:30.600361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:30.600393 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:30.603533 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:30.604226 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:30.604233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:30.604238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:30.604397 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:30.604786 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:30.604793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:30.604796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:30.605438 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:30.605733 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:30.606181 (+ 448us) server_negotiation.cc:299] Negotiation successful
0621 02:50:30.606552 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":418,"threads_started":1}
I20260621 02:50:30.608893 14701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bfb1eb517c46068d54040cf2777fe1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42717 } }
I20260621 02:50:30.609268 14701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:30.609391 14701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bfb1eb517c46068d54040cf2777fe1, State: Initialized, Role: FOLLOWER
I20260621 02:50:30.609961 14701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bfb1eb517c46068d54040cf2777fe1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42717 } }
I20260621 02:50:30.610190 14701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:30.610363 14701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:30.610551 14701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:30.611254 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:50:30.612733 14701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bfb1eb517c46068d54040cf2777fe1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42717 } }
I20260621 02:50:30.613305 14701 leader_election.cc:302] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04bfb1eb517c46068d54040cf2777fe1; no voters:
I20260621 02:50:30.613869 14701 leader_election.cc:288] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:30.614023 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:30.615231 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [term 1 LEADER]: Becoming Leader. State: Replica: 04bfb1eb517c46068d54040cf2777fe1, State: Running, Role: LEADER
I20260621 02:50:30.615911 14701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:30.615890 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bfb1eb517c46068d54040cf2777fe1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42717 } }
I20260621 02:50:30.626797 14708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bfb1eb517c46068d54040cf2777fe1. Latest consensus state: current_term: 1 leader_uuid: "04bfb1eb517c46068d54040cf2777fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bfb1eb517c46068d54040cf2777fe1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42717 } } }
I20260621 02:50:30.626901 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04bfb1eb517c46068d54040cf2777fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bfb1eb517c46068d54040cf2777fe1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42717 } } }
I20260621 02:50:30.627205 14708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:30.627259 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:30.637526 14720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:30.642670 14720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:30.663144 14720 catalog_manager.cc:1347] Generated new cluster ID: 9dcd550378cf461aa63159c7a2012e41
I20260621 02:50:30.663257 14720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:30.672881 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:30.665030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.11.109.62:42717)
0621 02:50:30.665213 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:30.665219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:30.665952 (+ 733us) server_negotiation.cc:407] Connection header received
0621 02:50:30.666178 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:30.666190 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:30.666255 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:30.666369 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:30.667386 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:30.668179 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:30.669537 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:30.669985 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:30.670116 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:30.671228 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:30.671234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:30.671237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:30.671976 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:30.672051 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:30.672084 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:30.672188 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:30.672378 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:50:30.672471 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260621 02:50:30.674291 14720 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:30.675486 14720 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:30.685494 14720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04bfb1eb517c46068d54040cf2777fe1: Generated new TSK 0
I20260621 02:50:30.686447 14720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:50:30.700846 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14634
I20260621 02:50:30.701987 14696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:30.790539 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14634
2026-06-21T02:50:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1873 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:30 dist-test-slave-bc52 krb5kdc[14736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:30 dist-test-slave-bc52 krb5kdc[14736](info): set up 2 sockets
Jun 21 02:50:30 dist-test-slave-bc52 krb5kdc[14736](info): commencing operation
krb5kdc: starting...
W20260621 02:50:31.865495 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:31 dist-test-slave-bc52 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:32.048053 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:42905
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:42905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:32.365828 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:32.366087 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:32.366267 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:32.373641 14749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:32.373735 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:32.373828 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:32.373921 14749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:32.374006 14749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:32.383666 14749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:42905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:42905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:32.384913 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:32.386183 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:32.397696 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:32.397667 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:32.397737 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:32.398715 14749 server_base.cc:1034] running on GCE node
I20260621 02:50:32.399371 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:32.400538 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:32.401778 14749 hybrid_clock.cc:648] HybridClock initialized: now 1782010232401746 us; error 59 us; skew 500 ppm
I20260621 02:50:32.402112 14749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:32 dist-test-slave-bc52 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:32.404945 14749 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:32.406386 14749 webserver.cc:460] Webserver started at http://127.11.109.62:45929/ using document root <none> and password file <none>
I20260621 02:50:32.407024 14749 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:32.407161 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:32.407509 14749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:32.410333 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "468d9dcdadd34c69b8ae49f07c001b67"
format_stamp: "Formatted at 2026-06-21 02:50:32 on dist-test-slave-bc52"
server_key: "83ebf18e5e13285b12de663fa677cf0b"
server_key_iv: "0ac334b544ef73091364976d0201348e"
server_key_version: "encryptionkey@0"
I20260621 02:50:32.411057 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "468d9dcdadd34c69b8ae49f07c001b67"
format_stamp: "Formatted at 2026-06-21 02:50:32 on dist-test-slave-bc52"
server_key: "83ebf18e5e13285b12de663fa677cf0b"
server_key_iv: "0ac334b544ef73091364976d0201348e"
server_key_version: "encryptionkey@0"
I20260621 02:50:32.415892 14749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:50:32.418951 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:32.420073 14749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260621 02:50:32.420231 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "468d9dcdadd34c69b8ae49f07c001b67"
format_stamp: "Formatted at 2026-06-21 02:50:32 on dist-test-slave-bc52"
server_key: "83ebf18e5e13285b12de663fa677cf0b"
server_key_iv: "0ac334b544ef73091364976d0201348e"
server_key_version: "encryptionkey@0"
I20260621 02:50:32.420406 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:32.440869 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:32.444519 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:32.444793 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:32.462342 14749 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:42905
I20260621 02:50:32.462373 14815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:42905 every 8 connection(s)
I20260621 02:50:32.463614 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:32.464113 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14749
I20260621 02:50:32.464321 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:32.465072 11700 external_mini_cluster.cc:1428] Setting key a9c1dba47439027138f44c158c5de521
I20260621 02:50:32.467746 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:32.477494 14816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67: Bootstrap starting.
Jun 21 02:50:32 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:32.480571 14816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:32.481637 14816 log.cc:824] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:32.484179 14816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67: No bootstrap required, opened a new log
I20260621 02:50:32.485677 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:32.466995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.11.109.62:42905)
0621 02:50:32.467692 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:32.467704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:32.467745 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:50:32.469393 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:32.469412 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:32.469749 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:32.470143 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:32.471511 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:32.472792 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:32.473999 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:32.474633 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:32.474826 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:32.478687 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:32.478695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:32.478708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:32.478752 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:32.481928 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:32.482577 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:32.482584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:32.482589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:32.482742 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:32.483113 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:32.483119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:32.483122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:32.483615 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:32.483922 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:32.484333 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 02:50:32.484716 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":320,"threads_started":1}
I20260621 02:50:32.492643 14816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d9dcdadd34c69b8ae49f07c001b67" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42905 } }
I20260621 02:50:32.493003 14816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:32.493109 14816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468d9dcdadd34c69b8ae49f07c001b67, State: Initialized, Role: FOLLOWER
I20260621 02:50:32.493696 14816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d9dcdadd34c69b8ae49f07c001b67" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42905 } }
I20260621 02:50:32.493911 14816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:32.494052 14816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:32.494230 14816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:32.496654 14816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d9dcdadd34c69b8ae49f07c001b67" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42905 } }
I20260621 02:50:32.497112 14816 leader_election.cc:302] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 468d9dcdadd34c69b8ae49f07c001b67; no voters:
I20260621 02:50:32.497718 14816 leader_election.cc:288] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:32.497880 14821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:32.498565 14821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [term 1 LEADER]: Becoming Leader. State: Replica: 468d9dcdadd34c69b8ae49f07c001b67, State: Running, Role: LEADER
I20260621 02:50:32.499061 14821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d9dcdadd34c69b8ae49f07c001b67" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42905 } }
I20260621 02:50:32.499949 14816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:32.501853 14823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 468d9dcdadd34c69b8ae49f07c001b67. Latest consensus state: current_term: 1 leader_uuid: "468d9dcdadd34c69b8ae49f07c001b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d9dcdadd34c69b8ae49f07c001b67" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42905 } } }
I20260621 02:50:32.502177 14823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:32.501945 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "468d9dcdadd34c69b8ae49f07c001b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "468d9dcdadd34c69b8ae49f07c001b67" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 42905 } } }
I20260621 02:50:32.502300 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:32.504334 14828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:32.508687 14828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:32.516338 14828 catalog_manager.cc:1347] Generated new cluster ID: 862febd2df68476aa1da34ed9dedc980
I20260621 02:50:32.516448 14828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:32.530853 14828 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:32.532112 14828 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:32.549626 14828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 468d9dcdadd34c69b8ae49f07c001b67: Generated new TSK 0
I20260621 02:50:32.550516 14828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:32.593282 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:42905
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:32.905639 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:32.905907 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:32.906067 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:32.913044 14842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:32.913144 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:32.913327 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:32.922948 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:42905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:32.924233 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:32.925609 14842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:32.938156 14847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:32.938270 14848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:32.938623 14850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:32.939105 14842 server_base.cc:1034] running on GCE node
I20260621 02:50:32.939741 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:32.940645 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:32.941884 14842 hybrid_clock.cc:648] HybridClock initialized: now 1782010232941832 us; error 46 us; skew 500 ppm
I20260621 02:50:32.942185 14842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:32 dist-test-slave-bc52 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:32.945302 14842 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:32.946758 14842 webserver.cc:460] Webserver started at http://127.11.109.1:38801/ using document root <none> and password file <none>
I20260621 02:50:32.947417 14842 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:32.947554 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:32.947897 14842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:32.950753 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "bd0a35cc5693486787aac64b2a9cacf8"
format_stamp: "Formatted at 2026-06-21 02:50:32 on dist-test-slave-bc52"
server_key: "331a8050a9d52fb70d647647f131d8e7"
server_key_iv: "8f30c5b7b3f34bbd2803d0cfe1cf0cd1"
server_key_version: "encryptionkey@0"
I20260621 02:50:32.951442 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "bd0a35cc5693486787aac64b2a9cacf8"
format_stamp: "Formatted at 2026-06-21 02:50:32 on dist-test-slave-bc52"
server_key: "331a8050a9d52fb70d647647f131d8e7"
server_key_iv: "8f30c5b7b3f34bbd2803d0cfe1cf0cd1"
server_key_version: "encryptionkey@0"
I20260621 02:50:32.956214 14842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:50:32.959444 14857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:32.960765 14842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:50:32.960961 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "bd0a35cc5693486787aac64b2a9cacf8"
format_stamp: "Formatted at 2026-06-21 02:50:32 on dist-test-slave-bc52"
server_key: "331a8050a9d52fb70d647647f131d8e7"
server_key_iv: "8f30c5b7b3f34bbd2803d0cfe1cf0cd1"
server_key_version: "encryptionkey@0"
I20260621 02:50:32.961156 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:33.006937 14842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:33.011121 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:33.011420 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:33.012394 14842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:33.013756 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:33.013865 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:33.014011 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:33.014083 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:33.046442 14842 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:37367
I20260621 02:50:33.046489 14969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:37367 every 8 connection(s)
I20260621 02:50:33.047701 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:33.051352 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14842
I20260621 02:50:33.051563 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:33.052330 11700 external_mini_cluster.cc:1428] Setting key 1930aa7a83ff059d274e5c6ddb1bf2cd
Jun 21 02:50:33 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010232, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:33.067689 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:33.050287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:58829 (local address 127.11.109.62:42905)
0621 02:50:33.050708 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:33.050713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:33.053244 (+ 2531us) server_negotiation.cc:407] Connection header received
0621 02:50:33.053769 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:33.053774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:33.053843 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:33.053935 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:33.055503 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.056213 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.057801 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.058200 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.058334 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:33.061896 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:33.061900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:33.061903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:33.061932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:33.064599 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:33.065478 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:33.065483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:33.065487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:33.065603 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:33.066034 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:33.066039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:33.066043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:33.066324 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:33.066471 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:33.067130 (+ 659us) server_negotiation.cc:299] Negotiation successful
0621 02:50:33.067346 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":219,"threads_started":1}
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:50:33.068631 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:33.050755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:42905 (local address 127.11.109.1:58829)
0621 02:50:33.051533 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:33.051579 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:33.053492 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:33.054250 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:33.054261 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:33.054786 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:33.055319 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:33.055336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.056388 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.056394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:33.057646 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:33.057658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.058423 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.058429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:33.058648 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:33.059304 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:33.059326 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:33.061680 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:33.064729 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:33.064738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:33.064754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:33.065357 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:33.065719 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:33.065724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:33.065728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:33.065915 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:33.066446 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:33.066454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:33.066766 (+ 312us) client_negotiation.cc:769] Sending connection context
0621 02:50:33.067032 (+ 266us) client_negotiation.cc:240] Negotiation successful
0621 02:50:33.067523 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":328,"threads_started":1}
I20260621 02:50:33.070099 14970 heartbeater.cc:347] Connected to a master server at 127.11.109.62:42905
I20260621 02:50:33.070397 14970 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:50:33.071214 14970 heartbeater.cc:511] Master 127.11.109.62:42905 requested a full tablet report, sending...
I20260621 02:50:33.073350 14781 ts_manager.cc:194] Registered new tserver with Master: bd0a35cc5693486787aac64b2a9cacf8 (127.11.109.1:37367)
I20260621 02:50:33.074747 14781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:58829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:33.092841 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:42905
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:33.402012 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:33.402249 14976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:33.402419 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:33.409736 14976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:33.409869 14976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:33.410051 14976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:33.419049 14976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:42905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:33.420243 14976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:33.421510 14976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:33.434196 14982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:33.434219 14981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:33.434861 14984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:33.435242 14976 server_base.cc:1034] running on GCE node
I20260621 02:50:33.435794 14976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:33.436610 14976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:33.437796 14976 hybrid_clock.cc:648] HybridClock initialized: now 1782010233437753 us; error 30 us; skew 500 ppm
I20260621 02:50:33.438153 14976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:33 dist-test-slave-bc52 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010233, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:33.441129 14976 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:33.442457 14976 webserver.cc:460] Webserver started at http://127.11.109.2:44751/ using document root <none> and password file <none>
I20260621 02:50:33.443096 14976 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:33.443238 14976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:33.443532 14976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:33.446290 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "e3517bc09c2748c192d527568e3964c6"
format_stamp: "Formatted at 2026-06-21 02:50:33 on dist-test-slave-bc52"
server_key: "3d58e2b190c6d460c57d90d62aec0b56"
server_key_iv: "c19c2c8325ed92f8dcda5da8c0f15b1f"
server_key_version: "encryptionkey@0"
I20260621 02:50:33.447068 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "e3517bc09c2748c192d527568e3964c6"
format_stamp: "Formatted at 2026-06-21 02:50:33 on dist-test-slave-bc52"
server_key: "3d58e2b190c6d460c57d90d62aec0b56"
server_key_iv: "c19c2c8325ed92f8dcda5da8c0f15b1f"
server_key_version: "encryptionkey@0"
I20260621 02:50:33.451963 14976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:50:33.455078 14991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:33.456290 14976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:33.456473 14976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "e3517bc09c2748c192d527568e3964c6"
format_stamp: "Formatted at 2026-06-21 02:50:33 on dist-test-slave-bc52"
server_key: "3d58e2b190c6d460c57d90d62aec0b56"
server_key_iv: "c19c2c8325ed92f8dcda5da8c0f15b1f"
server_key_version: "encryptionkey@0"
I20260621 02:50:33.456650 14976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:33.477555 14976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:33.482388 14976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:33.482685 14976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:33.483688 14976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:33.485001 14976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:33.485072 14976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:33.485154 14976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:33.485267 14976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 02:50:33.517726 14976 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:35389
I20260621 02:50:33.517776 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:35389 every 8 connection(s)
I20260621 02:50:33.519013 14976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:33.519511 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 14976
I20260621 02:50:33.519726 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:33.520334 11700 external_mini_cluster.cc:1428] Setting key 1772c89bbaecfe4aef57bafc00c6217c
Jun 21 02:50:33 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010233, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:50:33.542128 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:33.521860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:57323 (local address 127.11.109.62:42905)
0621 02:50:33.522022 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:33.522026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:33.523343 (+ 1317us) server_negotiation.cc:407] Connection header received
0621 02:50:33.525469 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:33.525474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:33.525565 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:33.525669 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:33.527134 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.528260 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.530991 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.531420 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.531519 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:33.535591 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:33.535598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:33.535604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:33.535654 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:33.539211 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:33.539914 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:33.539920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:33.539924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:33.540062 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:33.540529 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:33.540537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:33.540542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:33.540823 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:33.540937 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:33.541688 (+ 751us) server_negotiation.cc:299] Negotiation successful
0621 02:50:33.541856 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
I20260621 02:50:33.543143 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:33.522447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:42905 (local address 127.11.109.2:57323)
0621 02:50:33.523188 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:33.523230 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:33.525021 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:33.525915 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:33.525924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:33.526423 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:33.526972 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:33.526988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.528443 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.528447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:33.530847 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:33.530864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:33.531655 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:33.531664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:33.531994 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:33.532720 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:33.532758 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:33.535358 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:33.539351 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:33.539357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:33.539370 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:33.539772 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:33.540167 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:33.540170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:33.540172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:33.540402 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:33.540928 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:33.540933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:33.541337 (+ 404us) client_negotiation.cc:769] Sending connection context
0621 02:50:33.541598 (+ 261us) client_negotiation.cc:240] Negotiation successful
0621 02:50:33.541963 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":300,"threads_started":1}
I20260621 02:50:33.544622 15104 heartbeater.cc:347] Connected to a master server at 127.11.109.62:42905
I20260621 02:50:33.544950 15104 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:33.545862 15104 heartbeater.cc:511] Master 127.11.109.62:42905 requested a full tablet report, sending...
I20260621 02:50:33.547416 14781 ts_manager.cc:194] Registered new tserver with Master: e3517bc09c2748c192d527568e3964c6 (127.11.109.2:35389)
I20260621 02:50:33.548537 14781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:57323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:33.567888 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:42905
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:33.884302 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:33.884562 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:33.884652 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:33.891988 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:33.892088 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:33.892204 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:33.902046 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:42905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:33.903229 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:33.904593 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:33.917230 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:33.917279 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:33.917312 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:33.918222 15109 server_base.cc:1034] running on GCE node
I20260621 02:50:33.918788 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:33.919731 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:33.920926 15109 hybrid_clock.cc:648] HybridClock initialized: now 1782010233920885 us; error 35 us; skew 500 ppm
I20260621 02:50:33.921305 15109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:33 dist-test-slave-bc52 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010233, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:33.924467 15109 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:33.926040 15109 webserver.cc:460] Webserver started at http://127.11.109.3:39923/ using document root <none> and password file <none>
I20260621 02:50:33.926712 15109 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:33.926856 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:33.927155 15109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:33.930006 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "cd21dc80ba764ddb9e73c05de1c0ee2a"
format_stamp: "Formatted at 2026-06-21 02:50:33 on dist-test-slave-bc52"
server_key: "b4258c97010dfb1bbc781a799ce103a9"
server_key_iv: "6a8e733439199e1585080de365e18630"
server_key_version: "encryptionkey@0"
I20260621 02:50:33.930794 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "cd21dc80ba764ddb9e73c05de1c0ee2a"
format_stamp: "Formatted at 2026-06-21 02:50:33 on dist-test-slave-bc52"
server_key: "b4258c97010dfb1bbc781a799ce103a9"
server_key_iv: "6a8e733439199e1585080de365e18630"
server_key_version: "encryptionkey@0"
I20260621 02:50:33.935513 15109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:50:33.938791 15124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:33.939996 15109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:50:33.940182 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "cd21dc80ba764ddb9e73c05de1c0ee2a"
format_stamp: "Formatted at 2026-06-21 02:50:33 on dist-test-slave-bc52"
server_key: "b4258c97010dfb1bbc781a799ce103a9"
server_key_iv: "6a8e733439199e1585080de365e18630"
server_key_version: "encryptionkey@0"
I20260621 02:50:33.940367 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:33.963586 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:33.967499 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:33.967784 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:33.968690 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:33.970008 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:33.970113 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:33.970256 15109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:33.970330 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:34.002622 15109 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:41211
I20260621 02:50:34.002673 15236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:41211 every 8 connection(s)
I20260621 02:50:34.003912 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:34.004385 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15109
I20260621 02:50:34.004587 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:34.005209 11700 external_mini_cluster.cc:1428] Setting key 9e0fa6bd2b27d13196523053b6cb2983
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010233, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:34.022372 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:34.006495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:42289 (local address 127.11.109.62:42905)
0621 02:50:34.006695 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:34.006701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:34.007932 (+ 1231us) server_negotiation.cc:407] Connection header received
0621 02:50:34.009840 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:34.009844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:34.009909 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:34.009989 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:34.011328 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.012033 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.013536 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.013945 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.014038 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:34.017059 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:34.017064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:34.017067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:34.017106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:34.019802 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.020553 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.020557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.020559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.020685 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.021016 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.021019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.021021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.021326 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:34.021432 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:34.021967 (+ 535us) server_negotiation.cc:299] Negotiation successful
0621 02:50:34.022119 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:50:34.023293 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:34.007038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:42905 (local address 127.11.109.3:42289)
0621 02:50:34.007795 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:34.007842 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:34.009611 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:34.010167 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:34.010174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:34.010660 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:34.011184 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:34.011200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.012156 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.012160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:34.013423 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:34.013435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.014112 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.014116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:34.014294 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:34.014881 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:34.014902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:34.016886 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:34.019904 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:34.019912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:34.019925 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:34.020468 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:34.020774 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:34.020777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:34.020779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:34.020941 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:34.021418 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:34.021423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:34.021682 (+ 259us) client_negotiation.cc:769] Sending connection context
0621 02:50:34.021894 (+ 212us) client_negotiation.cc:240] Negotiation successful
0621 02:50:34.022218 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":284,"threads_started":1}
I20260621 02:50:34.024663 15237 heartbeater.cc:347] Connected to a master server at 127.11.109.62:42905
I20260621 02:50:34.024906 15237 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:34.025568 15237 heartbeater.cc:511] Master 127.11.109.62:42905 requested a full tablet report, sending...
I20260621 02:50:34.026912 14781 ts_manager.cc:194] Registered new tserver with Master: cd21dc80ba764ddb9e73c05de1c0ee2a (127.11.109.3:41211)
I20260621 02:50:34.027663 14781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:42289
I20260621 02:50:34.032092 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[14736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010234, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010234, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:34.077622 14970 heartbeater.cc:503] Master 127.11.109.62:42905 was elected leader, sending a full tablet report...
I20260621 02:50:34.079506 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:34.065500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.11.109.62:42905)
0621 02:50:34.065786 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:34.065794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:34.066183 (+ 389us) server_negotiation.cc:407] Connection header received
0621 02:50:34.066413 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:34.066418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:34.066480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:34.066586 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:34.067993 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.068736 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.070047 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.070479 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.070619 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:34.074403 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:34.074408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:34.074411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:34.074443 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:34.076830 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.077496 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.077502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.077507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.077639 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.078040 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.078050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.078052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.078452 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:34.078720 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:34.078995 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 02:50:34.079198 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260621 02:50:34.079995 14781 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54840
I20260621 02:50:34.081902 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14842
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010234, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:34.097540 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:34.083262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.11.109.1:37367)
0621 02:50:34.083872 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:34.083879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:34.083901 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:50:34.083949 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:34.083955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:34.084152 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:34.084316 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:34.085517 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.086488 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.087674 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.088225 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.088361 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:34.092151 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:34.092161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:34.092175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:34.092206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:34.094843 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.095506 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.095513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.095516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.095636 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.095998 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.096003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.096006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.096343 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:34.096659 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:34.096921 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 02:50:34.097209 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":298,"threads_started":1}
W20260621 02:50:34.098765 14965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55352
W20260621 02:50:34.099875 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14842: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:34.100093 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14842
I20260621 02:50:34.116370 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14976
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010234, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:34.131729 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:34.117231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.11.109.2:35389)
0621 02:50:34.117699 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:34.117706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:34.117730 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:50:34.117785 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:34.117791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:34.117999 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:34.118166 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:34.119383 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.120391 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.121769 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.122302 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.122443 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:34.126406 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:34.126416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:34.126430 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:34.126459 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:34.128989 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.129658 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.129665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.129668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.129787 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.130158 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.130167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.130169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.130487 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:34.130800 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:34.131108 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 02:50:34.131383 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
W20260621 02:50:34.132926 15099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34402
W20260621 02:50:34.134044 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:34.134168 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14976
I20260621 02:50:34.149367 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15109
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[14736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010234, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:34.164914 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:34.150195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.11.109.3:41211)
0621 02:50:34.150814 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:34.150821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:34.150848 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:50:34.150902 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:34.150908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:34.151108 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:34.151275 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:34.152537 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.153658 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.154876 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:34.155448 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:34.155584 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:34.159785 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:34.159796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:34.159810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:34.159848 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:34.162557 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.163172 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.163177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.163180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.163287 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:34.163587 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:34.163590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:34.163592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:34.163893 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:34.164107 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:34.164345 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 02:50:34.164533 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":323,"threads_started":1}
W20260621 02:50:34.166232 15232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42032
W20260621 02:50:34.167254 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15109: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:50:34.167413 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15109
I20260621 02:50:34.182437 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 14749
I20260621 02:50:34.183454 14811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:34.282805 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 14749
2026-06-21T02:50:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3498 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[15254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[15254](info): set up 2 sockets
Jun 21 02:50:34 dist-test-slave-bc52 krb5kdc[15254](info): commencing operation
krb5kdc: starting...
W20260621 02:50:35.369777 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:35 dist-test-slave-bc52 krb5kdc[15254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:35.550539 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46435
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:46435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:35.867866 15267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:35.868132 15267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:35.868297 15267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:35.875288 15267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:35.875407 15267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:35.875509 15267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:35.875607 15267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:35.875700 15267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:35.884989 15267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:46435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:35.886229 15267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:35.887447 15267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:35.899130 15275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:35.899169 15273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:35.899340 15272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:35.900017 15267 server_base.cc:1034] running on GCE node
I20260621 02:50:35.900768 15267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:35.902143 15267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:35.903345 15267 hybrid_clock.cc:648] HybridClock initialized: now 1782010235903297 us; error 36 us; skew 500 ppm
I20260621 02:50:35.903694 15267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:35 dist-test-slave-bc52 krb5kdc[15254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010235, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:35.906693 15267 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:35.908073 15267 webserver.cc:460] Webserver started at http://127.11.109.62:40751/ using document root <none> and password file <none>
I20260621 02:50:35.908708 15267 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:35.908855 15267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:35.909145 15267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:35.911918 15267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "c0f0dfb68ce24538b6441fb84e5b6399"
format_stamp: "Formatted at 2026-06-21 02:50:35 on dist-test-slave-bc52"
server_key: "75b0dfb1fa80ca3f83978f9046cd4bde"
server_key_iv: "55b9326a45c1838efe1fe7ec555ab5bc"
server_key_version: "encryptionkey@0"
I20260621 02:50:35.912652 15267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "c0f0dfb68ce24538b6441fb84e5b6399"
format_stamp: "Formatted at 2026-06-21 02:50:35 on dist-test-slave-bc52"
server_key: "75b0dfb1fa80ca3f83978f9046cd4bde"
server_key_iv: "55b9326a45c1838efe1fe7ec555ab5bc"
server_key_version: "encryptionkey@0"
I20260621 02:50:35.917385 15267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.006s
I20260621 02:50:35.920482 15282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:35.921722 15267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:50:35.921909 15267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "c0f0dfb68ce24538b6441fb84e5b6399"
format_stamp: "Formatted at 2026-06-21 02:50:35 on dist-test-slave-bc52"
server_key: "75b0dfb1fa80ca3f83978f9046cd4bde"
server_key_iv: "55b9326a45c1838efe1fe7ec555ab5bc"
server_key_version: "encryptionkey@0"
I20260621 02:50:35.922088 15267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:35.946360 15267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:35.950472 15267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:35.950745 15267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:35.968209 15267 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:46435
I20260621 02:50:35.968238 15333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:46435 every 8 connection(s)
I20260621 02:50:35.969480 15267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:35.972927 15334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:35.977161 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15267
I20260621 02:50:35.977396 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:35.978166 11700 external_mini_cluster.cc:1428] Setting key 5f9af59bd0aae015a9bda5ba6ce761f4
I20260621 02:50:35.983986 15334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399: Bootstrap starting.
I20260621 02:50:35.986920 15334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:35.987870 15334 log.cc:824] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:35.991123 15334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399: No bootstrap required, opened a new log
Jun 21 02:50:35 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:35.998692 15334 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46435 } }
I20260621 02:50:35.999142 15334 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:35.999251 15334 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f0dfb68ce24538b6441fb84e5b6399, State: Initialized, Role: FOLLOWER
I20260621 02:50:35.999850 15334 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46435 } }
I20260621 02:50:36.000079 15334 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:36.000211 15334 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:36.000248 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:35.980153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.11.109.62:46435)
0621 02:50:35.980853 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:35.980866 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:35.980907 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:50:35.982789 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:35.982813 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:35.983165 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:35.983539 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:35.984984 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:35.986347 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:35.987616 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:35.988374 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:35.988567 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:35.992917 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:35.992925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:35.992937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:35.992972 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:35.996102 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:35.996753 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:35.996758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:35.996764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:35.996935 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:35.997392 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:35.997397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:35.997402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:35.997889 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:35.998187 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:35.998589 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 02:50:35.999000 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":301,"threads_started":1}
I20260621 02:50:36.000402 15334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:36.002713 15334 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46435 } }
I20260621 02:50:36.003201 15334 leader_election.cc:302] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f0dfb68ce24538b6441fb84e5b6399; no voters:
I20260621 02:50:36.003674 15334 leader_election.cc:288] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:36.003808 15339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:36.005009 15339 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [term 1 LEADER]: Becoming Leader. State: Replica: c0f0dfb68ce24538b6441fb84e5b6399, State: Running, Role: LEADER
I20260621 02:50:36.005558 15339 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46435 } }
I20260621 02:50:36.005760 15334 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:36.008009 15341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f0dfb68ce24538b6441fb84e5b6399. Latest consensus state: current_term: 1 leader_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46435 } } }
I20260621 02:50:36.008363 15341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:36.008888 15340 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f0dfb68ce24538b6441fb84e5b6399" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46435 } } }
I20260621 02:50:36.009188 15340 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:36.010581 15346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:36.015848 15346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:36.023607 15346 catalog_manager.cc:1347] Generated new cluster ID: 0845a6256d86418894ed2943f4a7438a
I20260621 02:50:36.023707 15346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:36.038894 15346 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:36.040611 15346 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:36.048588 15346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f0dfb68ce24538b6441fb84e5b6399: Generated new TSK 0
I20260621 02:50:36.049401 15346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:36.100342 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46435
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:36.419394 15361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:36.419715 15361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:36.419912 15361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:36.427361 15361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:36.427469 15361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:36.427577 15361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:36.437217 15361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:36.438395 15361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:36.439585 15361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:36.452070 15367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:36.452303 15369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:36.452425 15366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:36.452888 15361 server_base.cc:1034] running on GCE node
I20260621 02:50:36.453538 15361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:36.454466 15361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:36.455726 15361 hybrid_clock.cc:648] HybridClock initialized: now 1782010236455688 us; error 45 us; skew 500 ppm
I20260621 02:50:36.456038 15361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:36 dist-test-slave-bc52 krb5kdc[15254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010236, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:36.459043 15361 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:36.460417 15361 webserver.cc:460] Webserver started at http://127.11.109.1:46785/ using document root <none> and password file <none>
I20260621 02:50:36.461086 15361 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:36.461252 15361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:36.461560 15361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:36.464325 15361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "8744bf7bd83945f49d0077232e66d2a6"
format_stamp: "Formatted at 2026-06-21 02:50:36 on dist-test-slave-bc52"
server_key: "07e9809eb4d77f75af87be45e80233c2"
server_key_iv: "2ceae8527659da8ad21cf5d9b34ccc05"
server_key_version: "encryptionkey@0"
I20260621 02:50:36.465070 15361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "8744bf7bd83945f49d0077232e66d2a6"
format_stamp: "Formatted at 2026-06-21 02:50:36 on dist-test-slave-bc52"
server_key: "07e9809eb4d77f75af87be45e80233c2"
server_key_iv: "2ceae8527659da8ad21cf5d9b34ccc05"
server_key_version: "encryptionkey@0"
I20260621 02:50:36.470661 15361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:50:36.474109 15376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:36.475560 15361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:50:36.475726 15361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "8744bf7bd83945f49d0077232e66d2a6"
format_stamp: "Formatted at 2026-06-21 02:50:36 on dist-test-slave-bc52"
server_key: "07e9809eb4d77f75af87be45e80233c2"
server_key_iv: "2ceae8527659da8ad21cf5d9b34ccc05"
server_key_version: "encryptionkey@0"
I20260621 02:50:36.475927 15361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:36.492893 15361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:36.496654 15361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:36.496933 15361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:36.497867 15361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:36.499156 15361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:36.499262 15361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:36.499413 15361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:36.499485 15361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:36.532825 15361 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:41695
I20260621 02:50:36.532913 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:41695 every 8 connection(s)
I20260621 02:50:36.534130 15361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:36.538918 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15361
I20260621 02:50:36.539131 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:36.539762 11700 external_mini_cluster.cc:1428] Setting key 2dc3aab49efd555f85ad946fc22819e8
Jun 21 02:50:36 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010236, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:36.554796 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:36.536543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:45815 (local address 127.11.109.62:46435)
0621 02:50:36.537000 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:36.537007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:36.538005 (+ 998us) server_negotiation.cc:407] Connection header received
0621 02:50:36.539800 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:36.539808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:36.539901 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:36.540027 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:36.541694 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:36.542746 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:36.544426 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:36.545143 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:36.545301 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:36.548883 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:36.548888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:36.548891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:36.548921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:36.551515 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:36.552414 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:36.552417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:36.552420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:36.552538 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:36.553018 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:36.553021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:36.553023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:36.553322 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:36.553444 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:36.554292 (+ 848us) server_negotiation.cc:299] Negotiation successful
0621 02:50:36.554467 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260621 02:50:36.556386 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:36.537129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46435 (local address 127.11.109.1:45815)
0621 02:50:36.537853 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:36.537890 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:36.539540 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:36.540218 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:36.540240 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:36.540861 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:36.541516 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:36.541545 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:36.542931 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:36.542935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:36.544272 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:36.544284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:36.545384 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:36.545389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:36.545589 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:36.546278 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:36.546305 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:36.548678 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:36.551654 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:36.551665 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:36.551684 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:36.552305 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:36.552653 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:36.552661 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:36.552667 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:36.552913 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:36.553455 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:36.553466 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:36.553928 (+ 462us) client_negotiation.cc:769] Sending connection context
0621 02:50:36.554253 (+ 325us) client_negotiation.cc:240] Negotiation successful
0621 02:50:36.554830 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":255,"threads_started":1}
I20260621 02:50:36.557960 15489 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46435
I20260621 02:50:36.558305 15489 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:50:36.559206 15489 heartbeater.cc:511] Master 127.11.109.62:46435 requested a full tablet report, sending...
I20260621 02:50:36.561403 15299 ts_manager.cc:194] Registered new tserver with Master: 8744bf7bd83945f49d0077232e66d2a6 (127.11.109.1:41695)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:50:36.563454 15299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:45815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:36.583971 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46435
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:36.895340 15495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:36.895635 15495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:36.895794 15495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:36.902824 15495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:36.902963 15495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:36.903149 15495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:36.912904 15495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:36.914129 15495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:36.915386 15495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:36.928539 15501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:36.928607 15500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:36.928630 15503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:36.929415 15495 server_base.cc:1034] running on GCE node
I20260621 02:50:36.930089 15495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:36.930994 15495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:36.932209 15495 hybrid_clock.cc:648] HybridClock initialized: now 1782010236932171 us; error 49 us; skew 500 ppm
I20260621 02:50:36.932515 15495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:36 dist-test-slave-bc52 krb5kdc[15254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010236, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:36.935506 15495 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:36.936875 15495 webserver.cc:460] Webserver started at http://127.11.109.2:34831/ using document root <none> and password file <none>
I20260621 02:50:36.937588 15495 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:36.937731 15495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:36.938061 15495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:36.940891 15495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "345f233178aa4af590ddebf455622462"
format_stamp: "Formatted at 2026-06-21 02:50:36 on dist-test-slave-bc52"
server_key: "9701764d31a7d3b4da400fde6ca04cd5"
server_key_iv: "f58161c71be8c704a07400f18f31d380"
server_key_version: "encryptionkey@0"
I20260621 02:50:36.941715 15495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "345f233178aa4af590ddebf455622462"
format_stamp: "Formatted at 2026-06-21 02:50:36 on dist-test-slave-bc52"
server_key: "9701764d31a7d3b4da400fde6ca04cd5"
server_key_iv: "f58161c71be8c704a07400f18f31d380"
server_key_version: "encryptionkey@0"
I20260621 02:50:36.946601 15495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 02:50:36.949898 15510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:36.951125 15495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:50:36.951288 15495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "345f233178aa4af590ddebf455622462"
format_stamp: "Formatted at 2026-06-21 02:50:36 on dist-test-slave-bc52"
server_key: "9701764d31a7d3b4da400fde6ca04cd5"
server_key_iv: "f58161c71be8c704a07400f18f31d380"
server_key_version: "encryptionkey@0"
I20260621 02:50:36.951506 15495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:36.987424 15495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:36.991822 15495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:36.992053 15495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:36.992936 15495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:36.994247 15495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:36.994320 15495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:36.994415 15495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:36.994459 15495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:37.027742 15495 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:39447
I20260621 02:50:37.027813 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:39447 every 8 connection(s)
I20260621 02:50:37.029021 15495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:37.030788 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15495
I20260621 02:50:37.030987 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:37.031524 11700 external_mini_cluster.cc:1428] Setting key bd2b5c671b8df99ef06a25f4468a66ff
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010236, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:50:37.050768 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.031566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:55569 (local address 127.11.109.62:46435)
0621 02:50:37.032273 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.032279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.033484 (+ 1205us) server_negotiation.cc:407] Connection header received
0621 02:50:37.035324 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.035332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.035425 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.035534 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.036997 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.038218 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.039689 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.040081 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.040177 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.044479 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.044485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.044489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.044522 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.047558 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.048333 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.048341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.048347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.048488 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.049046 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.049053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.049060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.049442 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.049596 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.050301 (+ 705us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.050474 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
I20260621 02:50:37.052175 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.032456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46435 (local address 127.11.109.2:55569)
0621 02:50:37.033312 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:37.033356 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:37.035058 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:37.035793 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:37.035801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:37.036291 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:37.036819 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:37.036835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.038361 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.038365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:37.039561 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:37.039572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.040315 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.040324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:37.040654 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.041494 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:37.041531 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:37.044233 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:37.047714 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:37.047726 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:37.047744 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:37.048202 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:37.048607 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:37.048614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:37.048618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:37.048893 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:37.049571 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:37.049577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:37.049975 (+ 398us) client_negotiation.cc:769] Sending connection context
0621 02:50:37.050235 (+ 260us) client_negotiation.cc:240] Negotiation successful
0621 02:50:37.050716 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":363,"threads_started":1}
I20260621 02:50:37.053661 15623 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46435
I20260621 02:50:37.053992 15623 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:37.054905 15623 heartbeater.cc:511] Master 127.11.109.62:46435 requested a full tablet report, sending...
I20260621 02:50:37.056502 15299 ts_manager.cc:194] Registered new tserver with Master: 345f233178aa4af590ddebf455622462 (127.11.109.2:39447)
I20260621 02:50:37.057643 15299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:55569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:37.074470 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46435
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:37.385906 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:37.386227 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:37.386448 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:37.393659 15628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:37.393767 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:37.393936 15628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:37.403568 15628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:37.404793 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:37.406175 15628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:37.420240 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:37.420291 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:37.420313 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:37.420861 15628 server_base.cc:1034] running on GCE node
I20260621 02:50:37.421520 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:37.422415 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:37.423646 15628 hybrid_clock.cc:648] HybridClock initialized: now 1782010237423598 us; error 42 us; skew 500 ppm
I20260621 02:50:37.423970 15628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:37.427384 15628 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:37.428711 15628 webserver.cc:460] Webserver started at http://127.11.109.3:40003/ using document root <none> and password file <none>
I20260621 02:50:37.429414 15628 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:37.429517 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:37.429771 15628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:37.432612 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "b33c61dedbcc44279e6f5b6495a4eac9"
format_stamp: "Formatted at 2026-06-21 02:50:37 on dist-test-slave-bc52"
server_key: "0a70aa0f7bd4bcb97f0b74e6bac24133"
server_key_iv: "a230f9ef0cac59f9aabe588ac67b3d1c"
server_key_version: "encryptionkey@0"
I20260621 02:50:37.433643 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "b33c61dedbcc44279e6f5b6495a4eac9"
format_stamp: "Formatted at 2026-06-21 02:50:37 on dist-test-slave-bc52"
server_key: "0a70aa0f7bd4bcb97f0b74e6bac24133"
server_key_iv: "a230f9ef0cac59f9aabe588ac67b3d1c"
server_key_version: "encryptionkey@0"
I20260621 02:50:37.438583 15628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:50:37.441903 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:37.443178 15628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:50:37.443364 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "b33c61dedbcc44279e6f5b6495a4eac9"
format_stamp: "Formatted at 2026-06-21 02:50:37 on dist-test-slave-bc52"
server_key: "0a70aa0f7bd4bcb97f0b74e6bac24133"
server_key_iv: "a230f9ef0cac59f9aabe588ac67b3d1c"
server_key_version: "encryptionkey@0"
I20260621 02:50:37.443573 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:37.490265 15628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:37.494710 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:37.494988 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:37.495958 15628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:37.497545 15628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:37.497663 15628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:37.497815 15628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:37.497874 15628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:37.550072 15628 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:43131
I20260621 02:50:37.550104 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:43131 every 8 connection(s)
I20260621 02:50:37.551416 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:37.551729 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15628
I20260621 02:50:37.551932 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:37.552506 11700 external_mini_cluster.cc:1428] Setting key 205a802551fe969355215ecc90e86b19
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:37.566443 15489 heartbeater.cc:503] Master 127.11.109.62:46435 was elected leader, sending a full tablet report...
I20260621 02:50:37.571758 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.554220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:50459 (local address 127.11.109.62:46435)
0621 02:50:37.554742 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.554748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.556116 (+ 1368us) server_negotiation.cc:407] Connection header received
0621 02:50:37.557999 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.558004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.558083 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.558213 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.559594 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.560354 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.561976 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.562391 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.562523 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.565805 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.565809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.565813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.565843 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.569229 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.569867 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.569872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.569874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.569985 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.570349 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.570352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.570353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.570591 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.570693 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.571282 (+ 589us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.571436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":238,"threads_started":1}
I20260621 02:50:37.572654 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.554995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46435 (local address 127.11.109.3:50459)
0621 02:50:37.555947 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:37.556000 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:37.557760 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:37.558433 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:37.558441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:37.558928 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:37.559433 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:37.559449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.560503 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.560509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:37.561840 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:37.561851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.562608 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.562615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:37.562840 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.563499 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:37.563523 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:37.565624 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:37.569353 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:37.569360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:37.569371 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:37.569763 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:37.570072 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:37.570075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:37.570078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:37.570254 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:37.570690 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:37.570697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:37.570983 (+ 286us) client_negotiation.cc:769] Sending connection context
0621 02:50:37.571212 (+ 229us) client_negotiation.cc:240] Negotiation successful
0621 02:50:37.571547 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":349,"threads_started":1}
I20260621 02:50:37.574030 15756 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46435
I20260621 02:50:37.574288 15756 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:37.574909 15756 heartbeater.cc:511] Master 127.11.109.62:46435 requested a full tablet report, sending...
I20260621 02:50:37.576288 15299 ts_manager.cc:194] Registered new tserver with Master: b33c61dedbcc44279e6f5b6495a4eac9 (127.11.109.3:43131)
I20260621 02:50:37.577067 15299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:50459
I20260621 02:50:37.581041 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:37.627707 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.612822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.11.109.1:41695)
0621 02:50:37.613583 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.613592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.613622 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:50:37.613696 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.613709 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.613922 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.614049 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.615348 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.616311 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.617556 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.618086 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.618185 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.622463 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.622474 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.622488 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.622521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.625156 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.625849 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.625855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.625857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.625981 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.626367 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.626371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.626373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.626681 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.626919 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.627145 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.627323 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":425,"threads_started":1}
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:37.644588 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.631726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.11.109.62:46435)
0621 02:50:37.631946 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.631954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.632159 (+ 205us) server_negotiation.cc:407] Connection header received
0621 02:50:37.632347 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.632353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.632426 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.632527 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.633821 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.634544 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.635734 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.636130 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.636274 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.639952 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.639956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.639959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.639989 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.642535 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.643088 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.643092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.643095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.643204 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.643499 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.643502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.643504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.643788 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.643952 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.644196 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.644367 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260621 02:50:37.645112 15299 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41410
I20260621 02:50:37.646133 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15361
I20260621 02:50:37.657608 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.647102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.11.109.1:41695)
0621 02:50:37.647258 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.647262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.647459 (+ 197us) server_negotiation.cc:407] Connection header received
0621 02:50:37.647576 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.647580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.647636 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.647712 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.648794 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.649496 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.650613 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.651008 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.651100 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.653134 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.653138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.653141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.653165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.655482 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.656043 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.656047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.656049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.656153 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.656468 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.656471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.656473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.656681 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.656844 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.657147 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.657390 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:50:37.658200 15484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:37.739607 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15361
I20260621 02:50:37.755637 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15495
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:37.770910 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.756503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.11.109.2:39447)
0621 02:50:37.757078 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.757085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.757111 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:50:37.757219 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.757233 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.757465 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.757611 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.758694 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.759659 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.760837 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.761472 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.761583 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.765514 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.765525 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.765539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.765570 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.768140 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.768804 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.768812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.768814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.768932 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.769353 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.769360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.769363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.769684 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.769989 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.770303 (+ 314us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.770558 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":307,"threads_started":1}
I20260621 02:50:37.772114 15618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:37.855837 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15495
I20260621 02:50:37.871589 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15628
Jun 21 02:50:37 dist-test-slave-bc52 krb5kdc[15254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:37.887820 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:37.872667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.11.109.3:43131)
0621 02:50:37.873401 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:37.873408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:37.873437 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:50:37.873499 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:37.873506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:37.873711 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:37.873855 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:37.875065 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.876052 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.877400 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:37.877931 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:37.878041 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:37.882708 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:37.882719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:37.882734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:37.882768 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:37.885453 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.886072 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.886079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.886081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.886199 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:37.886536 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:37.886540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:37.886542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:37.886885 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:37.887113 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:37.887349 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 02:50:37.887543 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":313,"threads_started":1}
I20260621 02:50:37.889150 15751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:37.970443 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15628
I20260621 02:50:37.990181 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15267
I20260621 02:50:37.991299 15329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:38.092073 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15267
2026-06-21T02:50:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3808 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:38 dist-test-slave-bc52 krb5kdc[15776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:38 dist-test-slave-bc52 krb5kdc[15776](info): set up 2 sockets
Jun 21 02:50:38 dist-test-slave-bc52 krb5kdc[15776](info): commencing operation
krb5kdc: starting...
W20260621 02:50:39.169553 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:39 dist-test-slave-bc52 krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:39.354817 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35393
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:39.663808 15789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:39.664058 15789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:39.664186 15789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:39.671054 15789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:39.671149 15789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:39.671248 15789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:39.671346 15789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:39.671450 15789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:39.680883 15789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:39.682083 15789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:39.683327 15789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:39.694759 15794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:39.694835 15795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:39.694867 15797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:39.695719 15789 server_base.cc:1034] running on GCE node
I20260621 02:50:39.696444 15789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:39.697901 15789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:39.699141 15789 hybrid_clock.cc:648] HybridClock initialized: now 1782010239699104 us; error 48 us; skew 500 ppm
I20260621 02:50:39.699478 15789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:39 dist-test-slave-bc52 krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010239, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:39.702457 15789 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:39.703528 15789 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:39.703658 15789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:39.703976 15789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:39.706826 15789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "1fab7e12fa574917a12515118942ed11"
format_stamp: "Formatted at 2026-06-21 02:50:39 on dist-test-slave-bc52"
server_key: "8c4ba5211c2100dcad611d057ee68542"
server_key_iv: "ba7cb8e237cbd926fa9cd15f12b6cf4b"
server_key_version: "encryptionkey@0"
I20260621 02:50:39.707598 15789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "1fab7e12fa574917a12515118942ed11"
format_stamp: "Formatted at 2026-06-21 02:50:39 on dist-test-slave-bc52"
server_key: "8c4ba5211c2100dcad611d057ee68542"
server_key_iv: "ba7cb8e237cbd926fa9cd15f12b6cf4b"
server_key_version: "encryptionkey@0"
I20260621 02:50:39.712522 15789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:50:39.715814 15803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:39.717067 15789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:39.717267 15789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "1fab7e12fa574917a12515118942ed11"
format_stamp: "Formatted at 2026-06-21 02:50:39 on dist-test-slave-bc52"
server_key: "8c4ba5211c2100dcad611d057ee68542"
server_key_iv: "ba7cb8e237cbd926fa9cd15f12b6cf4b"
server_key_version: "encryptionkey@0"
I20260621 02:50:39.717478 15789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:39.735922 15789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:39.739903 15789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:39.740150 15789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:39.757524 15789 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:35393
I20260621 02:50:39.757545 15854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:35393 every 8 connection(s)
I20260621 02:50:39.758421 15789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:39.761794 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:39.762774 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15789
I20260621 02:50:39.762998 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:39.763825 11700 external_mini_cluster.cc:1428] Setting key a6618f0b360b2af6874b372f54ccaf68
I20260621 02:50:39.775624 15855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11: Bootstrap starting.
Jun 21 02:50:39 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:39.779261 15855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:39.780676 15855 log.cc:824] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:39.785364 15855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11: No bootstrap required, opened a new log
I20260621 02:50:39.787750 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:39.765829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.11.109.62:35393)
0621 02:50:39.766607 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:39.766621 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:39.766659 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:50:39.768217 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:39.768238 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:39.768601 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:39.768927 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:39.770694 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:39.771949 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:39.773761 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:39.774383 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:39.774511 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:39.779072 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:39.779085 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:39.779104 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:39.779163 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:39.782586 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:39.783518 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:39.783528 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:39.783538 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:39.783727 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:39.784220 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:39.784228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:39.784231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:39.784961 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:39.785316 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:39.785904 (+ 588us) server_negotiation.cc:299] Negotiation successful
0621 02:50:39.786331 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260621 02:50:39.794381 15855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fab7e12fa574917a12515118942ed11" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35393 } }
I20260621 02:50:39.794732 15855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:39.794852 15855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fab7e12fa574917a12515118942ed11, State: Initialized, Role: FOLLOWER
I20260621 02:50:39.795375 15855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fab7e12fa574917a12515118942ed11" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35393 } }
I20260621 02:50:39.795606 15855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:39.795747 15855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:39.795921 15855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:39.798241 15855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fab7e12fa574917a12515118942ed11" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35393 } }
I20260621 02:50:39.798717 15855 leader_election.cc:302] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fab7e12fa574917a12515118942ed11; no voters:
I20260621 02:50:39.799278 15855 leader_election.cc:288] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:39.799438 15860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:39.800129 15860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [term 1 LEADER]: Becoming Leader. State: Replica: 1fab7e12fa574917a12515118942ed11, State: Running, Role: LEADER
I20260621 02:50:39.800654 15860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fab7e12fa574917a12515118942ed11" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35393 } }
I20260621 02:50:39.801426 15855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:39.803582 15862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fab7e12fa574917a12515118942ed11. Latest consensus state: current_term: 1 leader_uuid: "1fab7e12fa574917a12515118942ed11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fab7e12fa574917a12515118942ed11" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35393 } } }
I20260621 02:50:39.804181 15861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fab7e12fa574917a12515118942ed11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fab7e12fa574917a12515118942ed11" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35393 } } }
I20260621 02:50:39.804530 15862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:39.804651 15861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:39.805954 15869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:39.810413 15869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:39.818178 15869 catalog_manager.cc:1347] Generated new cluster ID: c64e2106fca9405580cd0d6007202558
I20260621 02:50:39.818279 15869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:39.846627 15869 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:39.848543 15869 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:39.873306 15869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11: Generated new TSK 0
I20260621 02:50:39.874140 15869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:39.925406 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35393
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:40.238102 15881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:40.238416 15881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:40.238582 15881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:40.245807 15881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:40.245949 15881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:40.246129 15881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:40.255599 15881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:40.256841 15881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:40.258183 15881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:40.270959 15889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:40.271025 15886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:40.270995 15887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:40.271780 15881 server_base.cc:1034] running on GCE node
I20260621 02:50:40.272404 15881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:40.273339 15881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:40.274559 15881 hybrid_clock.cc:648] HybridClock initialized: now 1782010240274515 us; error 55 us; skew 500 ppm
I20260621 02:50:40.274875 15881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:40 dist-test-slave-bc52 krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:40.277887 15881 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:40.278985 15881 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:40.279111 15881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:40.279419 15881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:40.282697 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "b5fd4337706641a39b7ce1c5fd54186a"
format_stamp: "Formatted at 2026-06-21 02:50:40 on dist-test-slave-bc52"
server_key: "b58cc1ddc313dc0eb7c8427f7c031168"
server_key_iv: "09c0c4676d5a9dab95fe83978376e8a3"
server_key_version: "encryptionkey@0"
I20260621 02:50:40.283854 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "b5fd4337706641a39b7ce1c5fd54186a"
format_stamp: "Formatted at 2026-06-21 02:50:40 on dist-test-slave-bc52"
server_key: "b58cc1ddc313dc0eb7c8427f7c031168"
server_key_iv: "09c0c4676d5a9dab95fe83978376e8a3"
server_key_version: "encryptionkey@0"
I20260621 02:50:40.290657 15881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 02:50:40.294020 15895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:40.295325 15881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:50:40.295486 15881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "b5fd4337706641a39b7ce1c5fd54186a"
format_stamp: "Formatted at 2026-06-21 02:50:40 on dist-test-slave-bc52"
server_key: "b58cc1ddc313dc0eb7c8427f7c031168"
server_key_iv: "09c0c4676d5a9dab95fe83978376e8a3"
server_key_version: "encryptionkey@0"
I20260621 02:50:40.295696 15881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:40.332492 15881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:40.336405 15881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:40.336666 15881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:40.337697 15881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:40.339130 15881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:40.339232 15881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:40.339358 15881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:40.339449 15881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:40.371366 15881 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:41679
I20260621 02:50:40.371452 16007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:41679 every 8 connection(s)
I20260621 02:50:40.372265 15881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:40.382217 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 15881
I20260621 02:50:40.382448 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:40.383098 11700 external_mini_cluster.cc:1428] Setting key 9fa6ebf7e939f6249de2685556293b42
Jun 21 02:50:40 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:40.392417 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:40.374821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:42839 (local address 127.11.109.62:35393)
0621 02:50:40.375253 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:40.375261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:40.376215 (+ 954us) server_negotiation.cc:407] Connection header received
0621 02:50:40.378018 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:40.378023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:40.378106 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:40.378221 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:40.379615 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.380321 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.381914 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.382343 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.382481 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:40.385932 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:40.385941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:40.385948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:40.385995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:40.389472 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:40.390147 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:40.390155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:40.390162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:40.390305 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:40.390679 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:40.390687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:40.390693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:40.391033 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:40.391143 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:40.392022 (+ 879us) server_negotiation.cc:299] Negotiation successful
0621 02:50:40.392185 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":203,"threads_started":1}
I20260621 02:50:40.393915 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:40.375415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35393 (local address 127.11.109.1:42839)
0621 02:50:40.376061 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:40.376103 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:40.377759 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:40.378432 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:40.378440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:40.378942 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:40.379458 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:40.379474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.380467 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.380478 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:40.381749 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:40.381760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.382560 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.382568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:40.382777 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:40.383447 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:40.383467 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:40.385724 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:40.389598 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:40.389604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:40.389615 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:40.390014 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:40.390407 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:40.390410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:40.390412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:40.390563 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:40.391167 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:40.391183 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:40.391596 (+ 413us) client_negotiation.cc:769] Sending connection context
0621 02:50:40.391935 (+ 339us) client_negotiation.cc:240] Negotiation successful
0621 02:50:40.392478 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":260,"threads_started":1}
I20260621 02:50:40.395431 16008 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35393
I20260621 02:50:40.395727 16008 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:40.396334 16008 heartbeater.cc:511] Master 127.11.109.62:35393 requested a full tablet report, sending...
I20260621 02:50:40.398232 15820 ts_manager.cc:194] Registered new tserver with Master: b5fd4337706641a39b7ce1c5fd54186a (127.11.109.1:41679)
I20260621 02:50:40.399785 15820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:42839
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:40.426539 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35393
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:40.733489 16014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:40.733822 16014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:40.733918 16014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:40.740851 16014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:40.740942 16014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:40.741108 16014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:40.750267 16014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:40.751402 16014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:40.752607 16014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:40.764936 16019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:40.764910 16020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:40.765378 16022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:40.765877 16014 server_base.cc:1034] running on GCE node
I20260621 02:50:40.766417 16014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:40.767340 16014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:40.768550 16014 hybrid_clock.cc:648] HybridClock initialized: now 1782010240768501 us; error 45 us; skew 500 ppm
I20260621 02:50:40.768849 16014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:40 dist-test-slave-bc52 krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:40.772215 16014 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:40.773566 16014 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:40.773681 16014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:40.774013 16014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:40.777994 16014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "efe8e04fac32485d9bb10fc36e3da7b6"
format_stamp: "Formatted at 2026-06-21 02:50:40 on dist-test-slave-bc52"
server_key: "c5f443b9d743a82f53795ca1478f7b36"
server_key_iv: "1cc2de2055ffb909bc7663f3161ab7af"
server_key_version: "encryptionkey@0"
I20260621 02:50:40.778836 16014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "efe8e04fac32485d9bb10fc36e3da7b6"
format_stamp: "Formatted at 2026-06-21 02:50:40 on dist-test-slave-bc52"
server_key: "c5f443b9d743a82f53795ca1478f7b36"
server_key_iv: "1cc2de2055ffb909bc7663f3161ab7af"
server_key_version: "encryptionkey@0"
I20260621 02:50:40.783859 16014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:50:40.787010 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:40.788192 16014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:40.788460 16014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "efe8e04fac32485d9bb10fc36e3da7b6"
format_stamp: "Formatted at 2026-06-21 02:50:40 on dist-test-slave-bc52"
server_key: "c5f443b9d743a82f53795ca1478f7b36"
server_key_iv: "1cc2de2055ffb909bc7663f3161ab7af"
server_key_version: "encryptionkey@0"
I20260621 02:50:40.788686 16014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:40.816211 16014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:40.820024 16014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:40.820297 16014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:40.821341 16014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:40.822655 16014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:40.822764 16014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:40.822903 16014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:40.822993 16014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:40.855187 16014 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:33351
I20260621 02:50:40.855263 16140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:33351 every 8 connection(s)
I20260621 02:50:40.856069 16014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:40.862898 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16014
I20260621 02:50:40.863092 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:40.863615 11700 external_mini_cluster.cc:1428] Setting key efde6993fd6982057953768b6da5511c
Jun 21 02:50:40 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010240, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:40.876103 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:40.858483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:46319 (local address 127.11.109.62:35393)
0621 02:50:40.858661 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:40.858665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:40.859786 (+ 1121us) server_negotiation.cc:407] Connection header received
0621 02:50:40.861606 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:40.861610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:40.861674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:40.861760 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:40.863458 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.864312 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.866298 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.866696 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.866789 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:40.870715 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:40.870720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:40.870723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:40.870753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:40.873266 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:40.873953 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:40.873958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:40.873962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:40.874092 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:40.874498 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:40.874503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:40.874506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:40.874755 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:40.874909 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:40.875568 (+ 659us) server_negotiation.cc:299] Negotiation successful
0621 02:50:40.875775 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:50:40.877069 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:40.858964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35393 (local address 127.11.109.2:46319)
0621 02:50:40.859627 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:40.859667 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:40.861365 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:40.861967 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:40.861975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:40.862474 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:40.863286 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:40.863311 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.864468 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.864475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:40.866165 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:40.866179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:40.866927 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:40.866937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:40.867254 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:40.867950 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:40.867983 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:40.870505 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:40.873396 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:40.873405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:40.873421 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:40.873827 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:40.874202 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:40.874207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:40.874211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:40.874382 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:40.874890 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:40.874898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:40.875209 (+ 311us) client_negotiation.cc:769] Sending connection context
0621 02:50:40.875470 (+ 261us) client_negotiation.cc:240] Negotiation successful
0621 02:50:40.875990 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":272,"threads_started":1}
I20260621 02:50:40.878445 16141 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35393
I20260621 02:50:40.878739 16141 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:40.879345 16141 heartbeater.cc:511] Master 127.11.109.62:35393 requested a full tablet report, sending...
I20260621 02:50:40.880762 15820 ts_manager.cc:194] Registered new tserver with Master: efe8e04fac32485d9bb10fc36e3da7b6 (127.11.109.2:33351)
I20260621 02:50:40.881588 15820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:46319
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:40.905669 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35393
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:41.209964 16146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:41.210206 16146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:41.210372 16146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:41.217140 16146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:41.217319 16146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:41.217504 16146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:41.226845 16146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:41.228003 16146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:41.229228 16146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:41.241357 16151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:41.241418 16152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:41.241446 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:41.242192 16146 server_base.cc:1034] running on GCE node
I20260621 02:50:41.242763 16146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:41.243598 16146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:41.244833 16146 hybrid_clock.cc:648] HybridClock initialized: now 1782010241244783 us; error 48 us; skew 500 ppm
I20260621 02:50:41.245134 16146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:41 dist-test-slave-bc52 krb5kdc[15776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:41.248080 16146 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:41.249250 16146 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:41.249378 16146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:41.249691 16146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:41.252389 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "88b248d220234228a0f38082f3796a51"
format_stamp: "Formatted at 2026-06-21 02:50:41 on dist-test-slave-bc52"
server_key: "edf4048689f84c133b3f3380d0d4d6dc"
server_key_iv: "f49aeed4776a0ba9129e348a6fd05b32"
server_key_version: "encryptionkey@0"
I20260621 02:50:41.253312 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "88b248d220234228a0f38082f3796a51"
format_stamp: "Formatted at 2026-06-21 02:50:41 on dist-test-slave-bc52"
server_key: "edf4048689f84c133b3f3380d0d4d6dc"
server_key_iv: "f49aeed4776a0ba9129e348a6fd05b32"
server_key_version: "encryptionkey@0"
I20260621 02:50:41.257925 16146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:50:41.261080 16160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:41.262305 16146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260621 02:50:41.262476 16146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "88b248d220234228a0f38082f3796a51"
format_stamp: "Formatted at 2026-06-21 02:50:41 on dist-test-slave-bc52"
server_key: "edf4048689f84c133b3f3380d0d4d6dc"
server_key_iv: "f49aeed4776a0ba9129e348a6fd05b32"
server_key_version: "encryptionkey@0"
I20260621 02:50:41.262684 16146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:41.276731 16146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:41.280400 16146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:41.280651 16146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:41.281596 16146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:41.282774 16146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:41.282902 16146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:41.283034 16146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:41.283126 16146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:41.319460 16146 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:45739
I20260621 02:50:41.319496 16272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:45739 every 8 connection(s)
I20260621 02:50:41.320291 16146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:41.320695 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16146
I20260621 02:50:41.320885 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:41.321486 11700 external_mini_cluster.cc:1428] Setting key c7de2eaca3d26639111519aafafefcf6
Jun 21 02:50:41 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010241, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:41.339795 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.323117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:55773 (local address 127.11.109.62:35393)
0621 02:50:41.323310 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.323314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.324675 (+ 1361us) server_negotiation.cc:407] Connection header received
0621 02:50:41.326453 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.326459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.326548 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.326654 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:41.328066 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.328792 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.330361 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.330762 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.330890 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.334162 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:41.334167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:41.334170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:41.334200 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:41.337375 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.337998 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.338002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.338004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.338122 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.338468 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.338471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.338473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.338708 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:41.338814 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.339392 (+ 578us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.339560 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:50:41.340662 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.323682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35393 (local address 127.11.109.3:55773)
0621 02:50:41.324504 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:41.324559 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:41.326201 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:41.326867 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:41.326875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:41.327365 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:41.327895 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.327914 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.328943 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.328948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.330218 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.330229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.330971 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.330980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.331217 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.331796 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:41.331817 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:41.333978 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:41.337494 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:41.337500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:41.337510 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:41.337896 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:41.338207 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:41.338210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:41.338212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:41.338372 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:41.338812 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:41.338817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:41.339088 (+ 271us) client_negotiation.cc:769] Sending connection context
0621 02:50:41.339324 (+ 236us) client_negotiation.cc:240] Negotiation successful
0621 02:50:41.339681 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":305,"threads_started":1}
I20260621 02:50:41.341979 16273 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35393
I20260621 02:50:41.342234 16273 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:41.342790 16273 heartbeater.cc:511] Master 127.11.109.62:35393 requested a full tablet report, sending...
I20260621 02:50:41.344297 15820 ts_manager.cc:194] Registered new tserver with Master: 88b248d220234228a0f38082f3796a51 (127.11.109.3:45739)
I20260621 02:50:41.345067 15820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:55773
I20260621 02:50:41.348745 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:50:41.364027 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.352970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.11.109.62:35393)
0621 02:50:41.353126 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.353130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.353597 (+ 467us) server_negotiation.cc:407] Connection header received
0621 02:50:41.353730 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.353733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.353784 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.353859 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:41.355143 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.355795 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.356939 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.357317 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.357399 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.359486 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:41.359491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:41.359494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:41.359532 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:41.362035 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.362567 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.362571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.362573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.362679 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.362992 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.362995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.362997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.363248 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:41.363411 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.363663 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.363802 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:50:41.367748 15820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:41.374423 15820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:41.397156 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.385564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:33351 (local address 127.0.0.1:53566)
0621 02:50:41.386653 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:41.386701 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:41.386896 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:41.388383 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:41.388393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:41.388434 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:41.388898 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.388913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.392243 (+ 3330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.392248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.394134 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.394146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.395985 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.395994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.396299 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.396385 (+ 86us) client_negotiation.cc:769] Sending connection context
0621 02:50:41.396556 (+ 171us) client_negotiation.cc:240] Negotiation successful
0621 02:50:41.396766 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":29,"thread_start_us":9250,"threads_started":1}
I20260621 02:50:41.397410 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.385928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.11.109.2:33351)
0621 02:50:41.387598 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.387610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.387645 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:50:41.387756 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.387767 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.388092 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.389146 (+ 1054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:41.389474 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.391530 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.394296 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.395755 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.395912 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.396144 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.396864 (+ 720us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.397042 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1299,"threads_started":1}
I20260621 02:50:41.401536 16076 tablet_service.cc:1467] Processing CreateTablet for tablet 92eafa4f83914f848bd59583265aff9e (DEFAULT_TABLE table=test-table [id=b8de9abebd1f4dd8823b80413d817d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:41.402729 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.386008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.11.109.1:41679)
0621 02:50:41.392126 (+ 6118us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.392136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.394803 (+ 2667us) server_negotiation.cc:407] Connection header received
0621 02:50:41.395098 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.395107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.395354 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.395523 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:41.396497 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.398001 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.400245 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.401546 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.401683 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.401888 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.402117 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.402329 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5958,"thread_start_us":5779,"threads_started":1}
I20260621 02:50:41.402871 16008 heartbeater.cc:503] Master 127.11.109.62:35393 was elected leader, sending a full tablet report...
I20260621 02:50:41.403440 16076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92eafa4f83914f848bd59583265aff9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:41.403604 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.385862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41679 (local address 127.0.0.1:37654)
0621 02:50:41.394686 (+ 8824us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:41.394707 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:41.394909 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:41.395781 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:41.395786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:41.395815 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:41.396336 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.396349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.398135 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.398140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.400099 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.400110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.401746 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.401752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.401904 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.401934 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:50:41.403004 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0621 02:50:41.403139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8706,"mutex_wait_us":46,"thread_start_us":302,"threads_started":1}
I20260621 02:50:41.404469 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.388393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.11.109.3:45739)
0621 02:50:41.392474 (+ 4081us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.392486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.395339 (+ 2853us) server_negotiation.cc:407] Connection header received
0621 02:50:41.395444 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.395453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.395662 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.395868 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:41.396854 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.398468 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.401029 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.402508 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.402626 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.403938 (+ 1312us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.404002 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.404144 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3910,"thread_start_us":3754,"threads_started":1}
I20260621 02:50:41.403704 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.387644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:45739 (local address 127.0.0.1:52856)
0621 02:50:41.395091 (+ 7447us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:41.395107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:41.395222 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:41.396148 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:41.396157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:41.396181 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:41.396646 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.396658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.398655 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.398659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.400865 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.400879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.402833 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.402839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.403013 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.403043 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:50:41.403121 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 02:50:41.403248 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7314,"mutex_wait_us":51,"thread_start_us":6777,"threads_started":1}
I20260621 02:50:41.407315 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 92eafa4f83914f848bd59583265aff9e (DEFAULT_TABLE table=test-table [id=b8de9abebd1f4dd8823b80413d817d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:41.408701 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92eafa4f83914f848bd59583265aff9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:41.408239 16208 tablet_service.cc:1467] Processing CreateTablet for tablet 92eafa4f83914f848bd59583265aff9e (DEFAULT_TABLE table=test-table [id=b8de9abebd1f4dd8823b80413d817d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:41.409513 16208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92eafa4f83914f848bd59583265aff9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:41.414921 16291 tablet_bootstrap.cc:492] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Bootstrap starting.
I20260621 02:50:41.417974 16291 tablet_bootstrap.cc:654] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:41.418936 16291 log.cc:824] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:41.421876 16291 tablet_bootstrap.cc:492] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: No bootstrap required, opened a new log
I20260621 02:50:41.422132 16291 ts_tablet_manager.cc:1397] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260621 02:50:41.423452 16293 tablet_bootstrap.cc:492] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Bootstrap starting.
I20260621 02:50:41.425007 16294 tablet_bootstrap.cc:492] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Bootstrap starting.
I20260621 02:50:41.426910 16293 tablet_bootstrap.cc:654] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:41.427917 16294 tablet_bootstrap.cc:654] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:41.428117 16293 log.cc:824] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:41.429051 16294 log.cc:824] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:41.431198 16293 tablet_bootstrap.cc:492] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: No bootstrap required, opened a new log
I20260621 02:50:41.431452 16293 ts_tablet_manager.cc:1397] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:50:41.431483 16294 tablet_bootstrap.cc:492] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: No bootstrap required, opened a new log
I20260621 02:50:41.431711 16294 ts_tablet_manager.cc:1397] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:50:41.432152 16291 raft_consensus.cc:348] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.432585 16291 raft_consensus.cc:374] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:41.432688 16291 raft_consensus.cc:729] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe8e04fac32485d9bb10fc36e3da7b6, State: Initialized, Role: FOLLOWER
I20260621 02:50:41.433359 16291 consensus_queue.cc:260] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.434415 16141 heartbeater.cc:503] Master 127.11.109.62:35393 was elected leader, sending a full tablet report...
I20260621 02:50:41.435087 16291 ts_tablet_manager.cc:1428] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260621 02:50:41.442718 16294 raft_consensus.cc:348] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.442718 16293 raft_consensus.cc:348] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.443130 16294 raft_consensus.cc:374] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:41.443130 16293 raft_consensus.cc:374] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:41.443234 16293 raft_consensus.cc:729] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b248d220234228a0f38082f3796a51, State: Initialized, Role: FOLLOWER
I20260621 02:50:41.443234 16294 raft_consensus.cc:729] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5fd4337706641a39b7ce1c5fd54186a, State: Initialized, Role: FOLLOWER
I20260621 02:50:41.443856 16293 consensus_queue.cc:260] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.443845 16294 consensus_queue.cc:260] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.444911 16273 heartbeater.cc:503] Master 127.11.109.62:35393 was elected leader, sending a full tablet report...
I20260621 02:50:41.445649 16293 ts_tablet_manager.cc:1428] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260621 02:50:41.445940 16294 ts_tablet_manager.cc:1428] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260621 02:50:41.460635 16297 raft_consensus.cc:482] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:41.460886 16297 raft_consensus.cc:504] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.462422 16297 leader_election.cc:288] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5fd4337706641a39b7ce1c5fd54186a (127.11.109.1:41679), 88b248d220234228a0f38082f3796a51 (127.11.109.3:45739)
I20260621 02:50:41.469947 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.462758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:60597 (local address 127.11.109.1:41679)
0621 02:50:41.462945 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.462950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.463713 (+ 763us) server_negotiation.cc:407] Connection header received
0621 02:50:41.463915 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.463920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.463984 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.464079 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:41.464956 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.466034 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.468141 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.468935 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.469053 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.469105 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.469542 (+ 437us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.469677 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:41.469980 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.462956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:50425 (local address 127.11.109.3:45739)
0621 02:50:41.463172 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.463181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.463713 (+ 532us) server_negotiation.cc:407] Connection header received
0621 02:50:41.463896 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.463901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.463984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.464079 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:41.464877 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.466050 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.468254 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.469055 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.469249 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.469323 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.469610 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.469730 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 02:50:41.470093 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.462762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41679 (local address 127.11.109.2:60597)
0621 02:50:41.463612 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:41.463629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:41.463775 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:41.464110 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:41.464115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:41.464151 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:41.464813 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.464825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.466232 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.466239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.467985 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.467998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.469147 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.469153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.469383 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.469407 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:50:41.469489 (+ 82us) client_negotiation.cc:240] Negotiation successful
0621 02:50:41.469632 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":176,"spinlock_wait_cycles":43136,"thread_start_us":368,"threads_started":1}
I20260621 02:50:41.470202 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.462754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:45739 (local address 127.11.109.2:50425)
0621 02:50:41.463580 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:41.463604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:41.463750 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:41.464110 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:41.464115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:41.464141 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:41.464730 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.464748 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.466182 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.466188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.468101 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:41.468117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.469322 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.469328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:41.469432 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.469453 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:50:41.469567 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 02:50:41.469685 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":77,"thread_start_us":410,"threads_started":1}
I20260621 02:50:41.471014 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eafa4f83914f848bd59583265aff9e" candidate_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5fd4337706641a39b7ce1c5fd54186a" is_pre_election: true
I20260621 02:50:41.471146 16228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eafa4f83914f848bd59583265aff9e" candidate_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b248d220234228a0f38082f3796a51" is_pre_election: true
I20260621 02:50:41.471504 15963 raft_consensus.cc:2454] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe8e04fac32485d9bb10fc36e3da7b6 in term 0.
I20260621 02:50:41.471644 16228 raft_consensus.cc:2454] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe8e04fac32485d9bb10fc36e3da7b6 in term 0.
I20260621 02:50:41.472199 16029 leader_election.cc:302] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5fd4337706641a39b7ce1c5fd54186a, efe8e04fac32485d9bb10fc36e3da7b6; no voters:
I20260621 02:50:41.472596 16297 raft_consensus.cc:2781] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:41.472741 16297 raft_consensus.cc:482] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:41.472858 16297 raft_consensus.cc:3037] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:41.475080 16297 raft_consensus.cc:504] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.475728 16297 leader_election.cc:288] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [CANDIDATE]: Term 1 election: Requested vote from peers b5fd4337706641a39b7ce1c5fd54186a (127.11.109.1:41679), 88b248d220234228a0f38082f3796a51 (127.11.109.3:45739)
I20260621 02:50:41.476336 16228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eafa4f83914f848bd59583265aff9e" candidate_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b248d220234228a0f38082f3796a51"
I20260621 02:50:41.476415 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92eafa4f83914f848bd59583265aff9e" candidate_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5fd4337706641a39b7ce1c5fd54186a"
I20260621 02:50:41.476550 16228 raft_consensus.cc:3037] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:41.476609 15963 raft_consensus.cc:3037] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:41.478749 16228 raft_consensus.cc:2454] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe8e04fac32485d9bb10fc36e3da7b6 in term 1.
I20260621 02:50:41.478781 15963 raft_consensus.cc:2454] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe8e04fac32485d9bb10fc36e3da7b6 in term 1.
I20260621 02:50:41.479209 16032 leader_election.cc:302] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 88b248d220234228a0f38082f3796a51, b5fd4337706641a39b7ce1c5fd54186a, efe8e04fac32485d9bb10fc36e3da7b6; no voters:
I20260621 02:50:41.479588 16297 raft_consensus.cc:2781] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:41.480011 16297 raft_consensus.cc:686] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 1 LEADER]: Becoming Leader. State: Replica: efe8e04fac32485d9bb10fc36e3da7b6, State: Running, Role: LEADER
I20260621 02:50:41.480515 16297 consensus_queue.cc:237] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } }
I20260621 02:50:41.484716 15820 catalog_manager.cc:5495] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to efe8e04fac32485d9bb10fc36e3da7b6 (127.11.109.2). New cstate: current_term: 1 leader_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efe8e04fac32485d9bb10fc36e3da7b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 } health_report { overall_health: UNKNOWN } } }
I20260621 02:50:41.509024 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.504202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.11.109.2:33351)
0621 02:50:41.504415 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.504431 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.504480 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:50:41.504643 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.504647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.504722 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.504835 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:41.505286 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.505953 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.507143 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.507524 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.507630 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.508013 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:41.508505 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:41.508605 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.508672 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.508730 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:50:41.516638 16228 raft_consensus.cc:1261] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 1 FOLLOWER]: Refusing update from remote peer efe8e04fac32485d9bb10fc36e3da7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:41.516638 15963 raft_consensus.cc:1261] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 1 FOLLOWER]: Refusing update from remote peer efe8e04fac32485d9bb10fc36e3da7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:41.517676 16297 consensus_queue.cc:1035] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b248d220234228a0f38082f3796a51" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:41.517980 16302 consensus_queue.cc:1035] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5fd4337706641a39b7ce1c5fd54186a" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:41.544512 15820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:41.544776 15820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:41.548043 15820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fab7e12fa574917a12515118942ed11: Sending DeleteTablet for 3 replicas of tablet 92eafa4f83914f848bd59583265aff9e
I20260621 02:50:41.549077 16076 tablet_service.cc:1514] Processing DeleteTablet for tablet 92eafa4f83914f848bd59583265aff9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:41 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:53566
I20260621 02:50:41.549456 15943 tablet_service.cc:1514] Processing DeleteTablet for tablet 92eafa4f83914f848bd59583265aff9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:41 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:37654
I20260621 02:50:41.550275 16208 tablet_service.cc:1514] Processing DeleteTablet for tablet 92eafa4f83914f848bd59583265aff9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:41 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:52856
I20260621 02:50:41.550634 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15881
I20260621 02:50:41.551137 16315 tablet_replica.cc:330] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: stopping tablet replica
I20260621 02:50:41.551568 16316 tablet_replica.cc:330] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: stopping tablet replica
I20260621 02:50:41.551940 16317 tablet_replica.cc:330] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: stopping tablet replica
I20260621 02:50:41.552142 16316 raft_consensus.cc:2229] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:41.552325 16317 raft_consensus.cc:2229] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:41.552734 16317 raft_consensus.cc:2258] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:41.552734 16316 raft_consensus.cc:2258] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:41.551708 16315 raft_consensus.cc:2229] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:41.553771 16315 raft_consensus.cc:2258] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:41.555488 16317 ts_tablet_manager.cc:1905] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:41.555611 16316 ts_tablet_manager.cc:1905] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:41.555600 16315 ts_tablet_manager.cc:1905] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:41.562282 16317 ts_tablet_manager.cc:1918] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:41.562513 16317 log.cc:1196] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/92eafa4f83914f848bd59583265aff9e
I20260621 02:50:41.562806 16315 ts_tablet_manager.cc:1918] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:41.563002 16315 log.cc:1196] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/92eafa4f83914f848bd59583265aff9e
I20260621 02:50:41.563360 16317 ts_tablet_manager.cc:1939] T 92eafa4f83914f848bd59583265aff9e P 88b248d220234228a0f38082f3796a51: Deleting consensus metadata
I20260621 02:50:41.563611 16315 ts_tablet_manager.cc:1939] T 92eafa4f83914f848bd59583265aff9e P efe8e04fac32485d9bb10fc36e3da7b6: Deleting consensus metadata
I20260621 02:50:41.565330 15805 catalog_manager.cc:4841] TS efe8e04fac32485d9bb10fc36e3da7b6 (127.11.109.2:33351): tablet 92eafa4f83914f848bd59583265aff9e (table test-table [id=b8de9abebd1f4dd8823b80413d817d88]) successfully deleted
I20260621 02:50:41.565449 15807 catalog_manager.cc:4841] TS 88b248d220234228a0f38082f3796a51 (127.11.109.3:45739): tablet 92eafa4f83914f848bd59583265aff9e (table test-table [id=b8de9abebd1f4dd8823b80413d817d88]) successfully deleted
Jun 21 02:50:41 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:41.569298 16316 ts_tablet_manager.cc:1918] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:41.569471 16316 log.cc:1196] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/92eafa4f83914f848bd59583265aff9e
I20260621 02:50:41.569988 16316 ts_tablet_manager.cc:1939] T 92eafa4f83914f848bd59583265aff9e P b5fd4337706641a39b7ce1c5fd54186a: Deleting consensus metadata
I20260621 02:50:41.571831 15804 catalog_manager.cc:4841] TS b5fd4337706641a39b7ce1c5fd54186a (127.11.109.1:41679): tablet 92eafa4f83914f848bd59583265aff9e (table test-table [id=b8de9abebd1f4dd8823b80413d817d88]) successfully deleted
I20260621 02:50:41.574755 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.552425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.11.109.1:41679)
0621 02:50:41.552652 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.552661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.553081 (+ 420us) server_negotiation.cc:407] Connection header received
0621 02:50:41.553347 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.553357 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.553417 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.553554 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:41.560533 (+ 6979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.561681 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.563563 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.564171 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.564281 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.568682 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:41.568698 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:41.568711 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:41.568741 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:41.571987 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.572750 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.572757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.572759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.572876 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.573306 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.573309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.573311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.573661 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:41.573885 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.574283 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.574451 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:50:41.575469 16003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:41.666214 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15881
I20260621 02:50:41.683152 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16014
Jun 21 02:50:41 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:41.699923 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.683959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.11.109.2:33351)
0621 02:50:41.684143 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.684149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.684170 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:50:41.684320 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.684326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.684416 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.684541 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:41.685771 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.687019 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.688207 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.688892 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.689023 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.693143 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:41.693164 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:41.693211 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:41.693256 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:41.697057 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.697844 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.697853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.697856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.698028 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.698381 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.698386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.698390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.698815 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:41.699147 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.699306 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.699514 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:50:41.700650 16136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:41.816267 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16014
I20260621 02:50:41.830647 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16146
Jun 21 02:50:41 dist-test-slave-bc52 krb5kdc[15776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:41.844980 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:41.831459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.11.109.3:45739)
0621 02:50:41.831639 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:41.831644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:41.831663 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:50:41.831845 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:41.831849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:41.831914 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:41.832002 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:41.833246 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.834053 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.835205 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:41.835562 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:41.835649 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:41.839737 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:41.839750 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:41.839766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:41.839807 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:41.842690 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.843308 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.843314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.843317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.843437 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:41.843740 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:41.843743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:41.843745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:41.844062 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:41.844254 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:41.844573 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 02:50:41.844725 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:50:41.845696 16268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:41.932688 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16146
I20260621 02:50:41.948894 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 15789
I20260621 02:50:41.950011 15850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:42.055651 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 15789
2026-06-21T02:50:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3962 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T02:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:42Z Disabled control of system clock
I20260621 02:50:42.118995 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:32875
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 02:50:42.426688 16326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:42.426962 16326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:42.427101 16326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:42.434260 16326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:42.434350 16326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:42.434441 16326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:42.434542 16326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:42.434625 16326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:42.443645 16326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:32875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:42.444837 16326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:42.446564 16326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:42.457783 16331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:42.457805 16332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:42.457964 16334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:42.458752 16326 server_base.cc:1034] running on GCE node
I20260621 02:50:42.459482 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:42.460772 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:42.461994 16326 hybrid_clock.cc:648] HybridClock initialized: now 1782010242461963 us; error 49 us; skew 500 ppm
I20260621 02:50:42.462349 16326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:42.464428 16326 webserver.cc:460] Webserver started at http://127.11.109.62:45701/ using document root <none> and password file <none>
I20260621 02:50:42.465088 16326 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:42.465240 16326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:42.465544 16326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:42.468313 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "64c38a4f021347b587b0d040728659d2"
format_stamp: "Formatted at 2026-06-21 02:50:42 on dist-test-slave-bc52"
server_key: "1911eeeac2e1c4d5aa1c6979ceb841f7"
server_key_iv: "a0ec20c808733e9bdb744d71ea036c5d"
server_key_version: "encryptionkey@0"
I20260621 02:50:42.469049 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "64c38a4f021347b587b0d040728659d2"
format_stamp: "Formatted at 2026-06-21 02:50:42 on dist-test-slave-bc52"
server_key: "1911eeeac2e1c4d5aa1c6979ceb841f7"
server_key_iv: "a0ec20c808733e9bdb744d71ea036c5d"
server_key_version: "encryptionkey@0"
I20260621 02:50:42.473738 16326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:50:42.476771 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:42.478053 16326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260621 02:50:42.478219 16326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "64c38a4f021347b587b0d040728659d2"
format_stamp: "Formatted at 2026-06-21 02:50:42 on dist-test-slave-bc52"
server_key: "1911eeeac2e1c4d5aa1c6979ceb841f7"
server_key_iv: "a0ec20c808733e9bdb744d71ea036c5d"
server_key_version: "encryptionkey@0"
I20260621 02:50:42.478420 16326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:42.483819 16326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:42.484555 16326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:42.484838 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:42.502784 16326 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:32875
I20260621 02:50:42.502823 16391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:32875 every 8 connection(s)
I20260621 02:50:42.504170 16326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:42.507699 16392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:42.514180 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16326
I20260621 02:50:42.514374 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:42.515024 11700 external_mini_cluster.cc:1428] Setting key 333bc4c0e8cbeeff80364353e4926bdd
I20260621 02:50:42.517771 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2: Bootstrap starting.
I20260621 02:50:42.521098 16392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:42.522303 16392 log.cc:824] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:42.523128 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:42.516821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.11.109.62:32875)
0621 02:50:42.517764 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:42.517774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:42.517813 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:50:42.519464 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:42.519483 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:42.519874 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:42.520207 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:42.521359 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:42.521366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:42.521378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:42.521408 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:42.521483 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:42.521503 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:42.521647 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:42.521789 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 02:50:42.522047 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":348,"threads_started":1}
I20260621 02:50:42.525163 16392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2: No bootstrap required, opened a new log
I20260621 02:50:42.533313 16392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c38a4f021347b587b0d040728659d2" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32875 } }
I20260621 02:50:42.533798 16392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:42.533950 16392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c38a4f021347b587b0d040728659d2, State: Initialized, Role: FOLLOWER
I20260621 02:50:42.534523 16392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c38a4f021347b587b0d040728659d2" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32875 } }
I20260621 02:50:42.534761 16392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:42.534924 16392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:42.535099 16392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:42.537139 16392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c38a4f021347b587b0d040728659d2" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32875 } }
I20260621 02:50:42.537706 16392 leader_election.cc:302] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64c38a4f021347b587b0d040728659d2; no voters:
I20260621 02:50:42.538293 16392 leader_election.cc:288] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:42.538435 16397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:42.539135 16397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [term 1 LEADER]: Becoming Leader. State: Replica: 64c38a4f021347b587b0d040728659d2, State: Running, Role: LEADER
I20260621 02:50:42.539795 16397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c38a4f021347b587b0d040728659d2" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32875 } }
I20260621 02:50:42.540366 16392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:42.542762 16399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c38a4f021347b587b0d040728659d2. Latest consensus state: current_term: 1 leader_uuid: "64c38a4f021347b587b0d040728659d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c38a4f021347b587b0d040728659d2" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32875 } } }
I20260621 02:50:42.542740 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64c38a4f021347b587b0d040728659d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c38a4f021347b587b0d040728659d2" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 32875 } } }
I20260621 02:50:42.543165 16399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:42.543167 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:42.545583 16405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:42.550422 16405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:42.558465 16405 catalog_manager.cc:1347] Generated new cluster ID: 0f3d573d633949fe97aa11ad263c2ffc
I20260621 02:50:42.558584 16405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:42.579483 16405 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:42.579852 16405 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:42.588101 16405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2: Generated new TSK 0
I20260621 02:50:42.588822 16405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:50:42.593325 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32875
--builtin_ntp_servers=127.11.109.20:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 02:50:42.906682 16416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:42.906929 16416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:42.907102 16416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:42.914206 16416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:42.914290 16416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:42.914494 16416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:42.923267 16416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:42.924476 16416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:42.926175 16416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:42.938678 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:42.938673 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:42.938916 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:42.939410 16416 server_base.cc:1034] running on GCE node
I20260621 02:50:42.939949 16416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:42.940837 16416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:42.942077 16416 hybrid_clock.cc:648] HybridClock initialized: now 1782010242942020 us; error 47 us; skew 500 ppm
I20260621 02:50:42.942381 16416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:42.944710 16416 webserver.cc:460] Webserver started at http://127.11.109.1:37009/ using document root <none> and password file <none>
I20260621 02:50:42.945482 16416 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:42.945595 16416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:42.945912 16416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:42.948602 16416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "996a5b7ec8d94518898b6f0ad0a5cefc"
format_stamp: "Formatted at 2026-06-21 02:50:42 on dist-test-slave-bc52"
server_key: "1025644c2fcf0ea4d0bedc20c941481b"
server_key_iv: "119f397939b299e836db11f56c0ec4f6"
server_key_version: "encryptionkey@0"
I20260621 02:50:42.949474 16416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "996a5b7ec8d94518898b6f0ad0a5cefc"
format_stamp: "Formatted at 2026-06-21 02:50:42 on dist-test-slave-bc52"
server_key: "1025644c2fcf0ea4d0bedc20c941481b"
server_key_iv: "119f397939b299e836db11f56c0ec4f6"
server_key_version: "encryptionkey@0"
I20260621 02:50:42.954219 16416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:50:42.957397 16430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:42.958597 16416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260621 02:50:42.958765 16416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "996a5b7ec8d94518898b6f0ad0a5cefc"
format_stamp: "Formatted at 2026-06-21 02:50:42 on dist-test-slave-bc52"
server_key: "1025644c2fcf0ea4d0bedc20c941481b"
server_key_iv: "119f397939b299e836db11f56c0ec4f6"
server_key_version: "encryptionkey@0"
I20260621 02:50:42.958971 16416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:42.964389 16416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:42.965128 16416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:42.965459 16416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:42.966300 16416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:42.967617 16416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:42.967736 16416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:42.967866 16416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:42.967955 16416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.000066 16416 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:35413
I20260621 02:50:43.000150 16542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:35413 every 8 connection(s)
I20260621 02:50:43.001438 16416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:43.009398 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.003915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:47601 (local address 127.11.109.62:32875)
0621 02:50:43.004085 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.004089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.005494 (+ 1405us) server_negotiation.cc:407] Connection header received
0621 02:50:43.007133 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.007138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.007211 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.007321 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.008621 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.008631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.008636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.008654 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.008692 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.008704 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.008839 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.009040 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.009108 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:50:43.010260 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.004417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:32875 (local address 127.11.109.1:47601)
0621 02:50:43.005340 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:43.005376 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:43.006862 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:43.007547 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:43.007554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:43.007643 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:43.008334 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:43.008336 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:43.008373 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:43.008457 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:43.008801 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:43.008814 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:43.008822 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:50:43.009108 (+ 286us) client_negotiation.cc:240] Negotiation successful
0621 02:50:43.009370 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":250,"threads_started":1}
I20260621 02:50:43.010547 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16416
I20260621 02:50:43.010739 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:43.011476 11700 external_mini_cluster.cc:1428] Setting key 3a0f4e6605e5248efa94f60ae36b6231
I20260621 02:50:43.011739 16543 heartbeater.cc:347] Connected to a master server at 127.11.109.62:32875
I20260621 02:50:43.012069 16543 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:43.012692 16543 heartbeater.cc:511] Master 127.11.109.62:32875 requested a full tablet report, sending...
I20260621 02:50:43.014518 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32875
--builtin_ntp_servers=127.11.109.20:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 02:50:43.014933 16357 ts_manager.cc:194] Registered new tserver with Master: 996a5b7ec8d94518898b6f0ad0a5cefc (127.11.109.1:35413)
W20260621 02:50:43.326870 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:43.327117 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:43.327203 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:43.334376 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:43.334473 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:43.334578 16546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:43.343853 16546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:43.345245 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:43.347056 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:43.360299 16551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:43.360267 16552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:43.360363 16554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:43.361147 16546 server_base.cc:1034] running on GCE node
I20260621 02:50:43.361820 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:43.362802 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:43.363999 16546 hybrid_clock.cc:648] HybridClock initialized: now 1782010243363979 us; error 40 us; skew 500 ppm
I20260621 02:50:43.364269 16546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:43.366371 16546 webserver.cc:460] Webserver started at http://127.11.109.2:43383/ using document root <none> and password file <none>
I20260621 02:50:43.366993 16546 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:43.367076 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:43.367337 16546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:43.370069 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "a0c736909d1b48059db08f8805a25a52"
format_stamp: "Formatted at 2026-06-21 02:50:43 on dist-test-slave-bc52"
server_key: "0fbd93195b76403922e1767b9f3a1c41"
server_key_iv: "7df5c3834e167f6c2e9d1fa4f8130b51"
server_key_version: "encryptionkey@0"
I20260621 02:50:43.370793 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "a0c736909d1b48059db08f8805a25a52"
format_stamp: "Formatted at 2026-06-21 02:50:43 on dist-test-slave-bc52"
server_key: "0fbd93195b76403922e1767b9f3a1c41"
server_key_iv: "7df5c3834e167f6c2e9d1fa4f8130b51"
server_key_version: "encryptionkey@0"
I20260621 02:50:43.375658 16546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:50:43.378963 16560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.380251 16546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:50:43.380381 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "a0c736909d1b48059db08f8805a25a52"
format_stamp: "Formatted at 2026-06-21 02:50:43 on dist-test-slave-bc52"
server_key: "0fbd93195b76403922e1767b9f3a1c41"
server_key_iv: "7df5c3834e167f6c2e9d1fa4f8130b51"
server_key_version: "encryptionkey@0"
I20260621 02:50:43.380545 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:43.386176 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:43.386868 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:43.387064 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:43.387905 16546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:43.389111 16546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:43.389263 16546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.389362 16546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:43.389420 16546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.421495 16546 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:44349
I20260621 02:50:43.421545 16672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:44349 every 8 connection(s)
I20260621 02:50:43.422814 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:43.423115 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16546
I20260621 02:50:43.423287 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:43.423828 11700 external_mini_cluster.cc:1428] Setting key 2597b933715c6a1308cb5c51b510366b
I20260621 02:50:43.426977 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32875
--builtin_ntp_servers=127.11.109.20:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 02:50:43.431468 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.425432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:58147 (local address 127.11.109.62:32875)
0621 02:50:43.425627 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.425637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.427436 (+ 1799us) server_negotiation.cc:407] Connection header received
0621 02:50:43.429099 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.429104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.429170 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.429322 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.430702 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.430708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.430713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.430741 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.430772 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.430792 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.430927 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.431115 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.431184 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:50:43.432385 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.426222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:32875 (local address 127.11.109.2:58147)
0621 02:50:43.427265 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:43.427322 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:43.428839 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:43.429552 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:43.429560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:43.429652 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:43.430389 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:43.430392 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:43.430430 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:43.430510 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:43.430888 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:43.430900 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:43.430910 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:50:43.431197 (+ 287us) client_negotiation.cc:240] Negotiation successful
0621 02:50:43.431467 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":307,"threads_started":1}
I20260621 02:50:43.433732 16673 heartbeater.cc:347] Connected to a master server at 127.11.109.62:32875
I20260621 02:50:43.434033 16673 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:43.434700 16673 heartbeater.cc:511] Master 127.11.109.62:32875 requested a full tablet report, sending...
I20260621 02:50:43.436187 16357 ts_manager.cc:194] Registered new tserver with Master: a0c736909d1b48059db08f8805a25a52 (127.11.109.2:44349)
W20260621 02:50:43.740701 16676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:43.740967 16676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:43.741108 16676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:43.748242 16676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:43.748325 16676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:43.748483 16676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:43.757081 16676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:43.758287 16676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:43.759891 16676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:43.772552 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:43.772768 16681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:43.773029 16682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:43.773376 16676 server_base.cc:1034] running on GCE node
I20260621 02:50:43.773880 16676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:43.774679 16676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:43.775869 16676 hybrid_clock.cc:648] HybridClock initialized: now 1782010243775818 us; error 44 us; skew 500 ppm
I20260621 02:50:43.776134 16676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:43.778302 16676 webserver.cc:460] Webserver started at http://127.11.109.3:44763/ using document root <none> and password file <none>
I20260621 02:50:43.778909 16676 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:43.778999 16676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:43.779311 16676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:43.782580 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "d0df5dba621b4d7a9db0970fe012e855"
format_stamp: "Formatted at 2026-06-21 02:50:43 on dist-test-slave-bc52"
server_key: "42607f5a075ef878983c2f99c6849b18"
server_key_iv: "96ec0a7ff1cf917677ab97ff5542926e"
server_key_version: "encryptionkey@0"
I20260621 02:50:43.783708 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "d0df5dba621b4d7a9db0970fe012e855"
format_stamp: "Formatted at 2026-06-21 02:50:43 on dist-test-slave-bc52"
server_key: "42607f5a075ef878983c2f99c6849b18"
server_key_iv: "96ec0a7ff1cf917677ab97ff5542926e"
server_key_version: "encryptionkey@0"
I20260621 02:50:43.790300 16676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 02:50:43.793563 16690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.794760 16676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:50:43.794935 16676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "d0df5dba621b4d7a9db0970fe012e855"
format_stamp: "Formatted at 2026-06-21 02:50:43 on dist-test-slave-bc52"
server_key: "42607f5a075ef878983c2f99c6849b18"
server_key_iv: "96ec0a7ff1cf917677ab97ff5542926e"
server_key_version: "encryptionkey@0"
I20260621 02:50:43.795147 16676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:43.800525 16676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:43.801326 16676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:43.801575 16676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:43.802423 16676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:43.803700 16676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:43.803810 16676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.803929 16676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:43.804026 16676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:43.836002 16676 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:41759
I20260621 02:50:43.836092 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:41759 every 8 connection(s)
I20260621 02:50:43.837307 16676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:43.845031 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.839682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:52915 (local address 127.11.109.62:32875)
0621 02:50:43.839912 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.839919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.841391 (+ 1472us) server_negotiation.cc:407] Connection header received
0621 02:50:43.843055 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.843059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.843110 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.843197 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.844405 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.844410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.844413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.844429 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.844460 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.844471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.844569 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.844781 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.844836 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:50:43.845818 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16676
I20260621 02:50:43.845978 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:43.846354 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.840215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:32875 (local address 127.11.109.3:52915)
0621 02:50:43.841213 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:43.841259 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:43.842818 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:43.843392 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:43.843400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:43.843488 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:43.844126 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:43.844128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:43.844165 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:43.844244 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:43.844560 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:43.844567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:43.844572 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:50:43.844824 (+ 252us) client_negotiation.cc:240] Negotiation successful
0621 02:50:43.845068 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":290,"threads_started":1}
I20260621 02:50:43.846556 11700 external_mini_cluster.cc:1428] Setting key 684a55702d74d252b21605b3ecaeb132
I20260621 02:50:43.847826 16803 heartbeater.cc:347] Connected to a master server at 127.11.109.62:32875
I20260621 02:50:43.848089 16803 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:43.848780 16803 heartbeater.cc:511] Master 127.11.109.62:32875 requested a full tablet report, sending...
I20260621 02:50:43.850194 16357 ts_manager.cc:194] Registered new tserver with Master: d0df5dba621b4d7a9db0970fe012e855 (127.11.109.3:41759)
I20260621 02:50:43.861932 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:50:43.869057 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.866413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.11.109.62:32875)
0621 02:50:43.866616 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.866624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.867140 (+ 516us) server_negotiation.cc:407] Connection header received
0621 02:50:43.867353 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.867358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.867420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.867549 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.868397 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.868403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.868408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.868426 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.868465 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.868476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.868598 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.868731 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.868790 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 02:50:43.872767 16357 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:43.879280 16357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:43.900657 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.891583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:41759 (local address 127.0.0.1:44982)
0621 02:50:43.892637 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:43.892683 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:43.892849 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:43.896510 (+ 3661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:43.896521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:43.896636 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:43.897661 (+ 1025us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:43.897667 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:43.897705 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:43.897812 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:43.898491 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:43.898505 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:43.898517 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 02:50:43.898728 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 02:50:43.898840 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"threads_started":1}
I20260621 02:50:43.901619 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.890757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.11.109.3:41759)
0621 02:50:43.891521 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.891529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.893138 (+ 1609us) server_negotiation.cc:407] Connection header received
0621 02:50:43.893417 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.893430 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.893712 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.893873 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.898091 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.898104 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.898124 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.898148 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.898223 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.898250 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.898436 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.900945 (+ 2509us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.901078 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":449,"threads_started":1}
I20260621 02:50:43.907413 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.890897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.11.109.2:44349)
0621 02:50:43.891380 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.891389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.902115 (+ 10726us) server_negotiation.cc:407] Connection header received
0621 02:50:43.902366 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.902378 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.902640 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.902789 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.906288 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.906301 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.906320 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.906347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.906424 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.906449 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.906623 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.906887 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.907018 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":510,"threads_started":1}
I20260621 02:50:43.908156 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.891600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:44349 (local address 127.0.0.1:50230)
0621 02:50:43.901988 (+ 10388us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:43.902010 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:43.902205 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:43.905274 (+ 3069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:43.905281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:43.905332 (+ 51us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:43.905993 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:43.905999 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:43.906022 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:43.906082 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:43.906722 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:43.906730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:43.906737 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:50:43.907851 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0621 02:50:43.907904 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10244,"thread_start_us":15657,"threads_started":1}
I20260621 02:50:43.909447 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.892090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.11.109.1:35413)
0621 02:50:43.905880 (+ 13790us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:43.905891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:43.905919 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:50:43.905992 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:43.905999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:43.906190 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:43.906328 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:43.908637 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:43.908645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:43.908656 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:43.908682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:43.908735 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:43.908753 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:43.908884 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:43.909023 (+ 139us) server_negotiation.cc:299] Negotiation successful
0621 02:50:43.909120 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13612,"thread_start_us":13488,"threads_started":1}
I20260621 02:50:43.909528 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:43.891804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52056)
0621 02:50:43.904698 (+ 12894us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:43.904721 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:43.904835 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:43.907226 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:43.907230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:43.907255 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:43.908409 (+ 1154us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:43.908414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:43.908432 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:43.908479 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:43.908875 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:43.908879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:43.908884 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:50:43.909020 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 02:50:43.909069 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12720,"thread_start_us":346,"threads_started":1}
I20260621 02:50:43.909505 16738 tablet_service.cc:1467] Processing CreateTablet for tablet 20e2efdd398c48a5be0e2cf74ec552ec (DEFAULT_TABLE table=test-table [id=ca59048083f24a918f9204ed133532b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:43.910895 16738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e2efdd398c48a5be0e2cf74ec552ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:43.912298 16608 tablet_service.cc:1467] Processing CreateTablet for tablet 20e2efdd398c48a5be0e2cf74ec552ec (DEFAULT_TABLE table=test-table [id=ca59048083f24a918f9204ed133532b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:43.912823 16478 tablet_service.cc:1467] Processing CreateTablet for tablet 20e2efdd398c48a5be0e2cf74ec552ec (DEFAULT_TABLE table=test-table [id=ca59048083f24a918f9204ed133532b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:43.913646 16608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e2efdd398c48a5be0e2cf74ec552ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:43.914068 16478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e2efdd398c48a5be0e2cf74ec552ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:43.921459 16821 tablet_bootstrap.cc:492] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Bootstrap starting.
I20260621 02:50:43.924746 16821 tablet_bootstrap.cc:654] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:43.925757 16821 log.cc:824] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:43.928181 16821 tablet_bootstrap.cc:492] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: No bootstrap required, opened a new log
I20260621 02:50:43.928468 16821 ts_tablet_manager.cc:1397] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:50:43.928963 16824 tablet_bootstrap.cc:492] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Bootstrap starting.
I20260621 02:50:43.932296 16824 tablet_bootstrap.cc:654] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:43.933490 16824 log.cc:824] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:43.935230 16823 tablet_bootstrap.cc:492] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Bootstrap starting.
I20260621 02:50:43.936364 16824 tablet_bootstrap.cc:492] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: No bootstrap required, opened a new log
I20260621 02:50:43.936609 16824 ts_tablet_manager.cc:1397] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:50:43.937721 16823 tablet_bootstrap.cc:654] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:43.938589 16823 log.cc:824] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:43.938705 16821 raft_consensus.cc:348] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:43.939201 16821 raft_consensus.cc:374] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:43.939316 16821 raft_consensus.cc:729] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0df5dba621b4d7a9db0970fe012e855, State: Initialized, Role: FOLLOWER
I20260621 02:50:43.939903 16821 consensus_queue.cc:260] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:43.941013 16803 heartbeater.cc:503] Master 127.11.109.62:32875 was elected leader, sending a full tablet report...
I20260621 02:50:43.941603 16821 ts_tablet_manager.cc:1428] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260621 02:50:43.942250 16823 tablet_bootstrap.cc:492] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: No bootstrap required, opened a new log
I20260621 02:50:43.942556 16823 ts_tablet_manager.cc:1397] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260621 02:50:43.946993 16824 raft_consensus.cc:348] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:43.947387 16824 raft_consensus.cc:374] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:43.947496 16824 raft_consensus.cc:729] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996a5b7ec8d94518898b6f0ad0a5cefc, State: Initialized, Role: FOLLOWER
I20260621 02:50:43.948076 16824 consensus_queue.cc:260] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:43.949043 16543 heartbeater.cc:503] Master 127.11.109.62:32875 was elected leader, sending a full tablet report...
I20260621 02:50:43.949678 16824 ts_tablet_manager.cc:1428] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260621 02:50:43.951638 16823 raft_consensus.cc:348] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:43.951960 16823 raft_consensus.cc:374] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:43.952057 16823 raft_consensus.cc:729] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c736909d1b48059db08f8805a25a52, State: Initialized, Role: FOLLOWER
I20260621 02:50:43.952575 16823 consensus_queue.cc:260] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:43.953555 16673 heartbeater.cc:503] Master 127.11.109.62:32875 was elected leader, sending a full tablet report...
I20260621 02:50:43.954056 16823 ts_tablet_manager.cc:1428] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260621 02:50:44.003551 16544 tablet.cc:2369] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:50:44.088919 16804 tablet.cc:2369] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:50:44.174885 16674 tablet.cc:2369] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:50:44.261634 16829 raft_consensus.cc:482] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:44.261934 16829 raft_consensus.cc:504] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:44.263733 16829 leader_election.cc:288] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0df5dba621b4d7a9db0970fe012e855 (127.11.109.3:41759), 996a5b7ec8d94518898b6f0ad0a5cefc (127.11.109.1:35413)
I20260621 02:50:44.267472 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.264090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:50291 (local address 127.11.109.1:35413)
0621 02:50:44.264282 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:44.264288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:44.265455 (+ 1167us) server_negotiation.cc:407] Connection header received
0621 02:50:44.265633 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:44.265639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:44.265700 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:44.265809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:44.266826 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:44.266831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:44.266836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:44.266855 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:44.266893 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:44.266905 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:44.267024 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:44.267124 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 02:50:44.267186 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:50:44.267557 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.264121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:36629 (local address 127.11.109.3:41759)
0621 02:50:44.264302 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:44.264307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:44.265403 (+ 1096us) server_negotiation.cc:407] Connection header received
0621 02:50:44.265602 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:44.265606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:44.265652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:44.265756 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:44.266934 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:44.266939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:44.266948 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:44.266961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:44.266989 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:44.267000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:44.267149 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:44.267224 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 02:50:44.267284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:50:44.267668 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.264328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.11.109.2:50291)
0621 02:50:44.265353 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.265371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:44.265493 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:44.265830 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:44.265835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:44.265867 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:44.266616 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:44.266618 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:44.266650 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:44.266689 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:44.266998 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:44.267002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:44.267007 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:50:44.267154 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 02:50:44.267201 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":537,"threads_started":1}
I20260621 02:50:44.267799 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.264273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:41759 (local address 127.11.109.2:36629)
0621 02:50:44.265285 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.265307 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:44.265451 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:44.265783 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:44.265797 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:44.265820 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:44.266734 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:50:44.266737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:44.266749 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:50:44.266782 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:44.267114 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:44.267118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:44.267124 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:50:44.267271 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 02:50:44.267324 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":604,"threads_started":1}
I20260621 02:50:44.268731 16758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e2efdd398c48a5be0e2cf74ec552ec" candidate_uuid: "a0c736909d1b48059db08f8805a25a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0df5dba621b4d7a9db0970fe012e855" is_pre_election: true
I20260621 02:50:44.269047 16498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e2efdd398c48a5be0e2cf74ec552ec" candidate_uuid: "a0c736909d1b48059db08f8805a25a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" is_pre_election: true
I20260621 02:50:44.269326 16758 raft_consensus.cc:2454] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c736909d1b48059db08f8805a25a52 in term 0.
I20260621 02:50:44.269613 16498 raft_consensus.cc:2454] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0c736909d1b48059db08f8805a25a52 in term 0.
I20260621 02:50:44.269999 16562 leader_election.cc:302] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c736909d1b48059db08f8805a25a52, d0df5dba621b4d7a9db0970fe012e855; no voters:
I20260621 02:50:44.270397 16829 raft_consensus.cc:2781] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:44.270535 16829 raft_consensus.cc:482] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:44.270655 16829 raft_consensus.cc:3037] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:44.272755 16829 raft_consensus.cc:504] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:44.273389 16829 leader_election.cc:288] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [CANDIDATE]: Term 1 election: Requested vote from peers d0df5dba621b4d7a9db0970fe012e855 (127.11.109.3:41759), 996a5b7ec8d94518898b6f0ad0a5cefc (127.11.109.1:35413)
I20260621 02:50:44.273764 16758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e2efdd398c48a5be0e2cf74ec552ec" candidate_uuid: "a0c736909d1b48059db08f8805a25a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0df5dba621b4d7a9db0970fe012e855"
I20260621 02:50:44.273875 16498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e2efdd398c48a5be0e2cf74ec552ec" candidate_uuid: "a0c736909d1b48059db08f8805a25a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc"
I20260621 02:50:44.273963 16758 raft_consensus.cc:3037] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:44.274055 16498 raft_consensus.cc:3037] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:44.277232 16498 raft_consensus.cc:2454] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c736909d1b48059db08f8805a25a52 in term 1.
I20260621 02:50:44.277228 16758 raft_consensus.cc:2454] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0c736909d1b48059db08f8805a25a52 in term 1.
I20260621 02:50:44.277594 16564 leader_election.cc:302] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996a5b7ec8d94518898b6f0ad0a5cefc, a0c736909d1b48059db08f8805a25a52; no voters:
I20260621 02:50:44.277882 16829 raft_consensus.cc:2781] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:44.278273 16829 raft_consensus.cc:686] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 1 LEADER]: Becoming Leader. State: Replica: a0c736909d1b48059db08f8805a25a52, State: Running, Role: LEADER
I20260621 02:50:44.278702 16829 consensus_queue.cc:237] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } }
I20260621 02:50:44.283610 16357 catalog_manager.cc:5495] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0c736909d1b48059db08f8805a25a52 (127.11.109.2). New cstate: current_term: 1 leader_uuid: "a0c736909d1b48059db08f8805a25a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0c736909d1b48059db08f8805a25a52" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 } health_report { overall_health: UNKNOWN } } }
I20260621 02:50:44.352135 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.350210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.11.109.2:44349)
0621 02:50:44.350387 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:44.350391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:44.350411 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:50:44.350574 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:44.350578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:44.350656 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:44.350740 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:44.351522 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:44.351526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:44.351529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:44.351543 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:44.351573 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:44.351583 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:44.351681 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:44.351804 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 02:50:44.351849 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:50:44.360365 16498 raft_consensus.cc:1261] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 1 FOLLOWER]: Refusing update from remote peer a0c736909d1b48059db08f8805a25a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:44.360373 16758 raft_consensus.cc:1261] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 1 FOLLOWER]: Refusing update from remote peer a0c736909d1b48059db08f8805a25a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:44.361521 16832 consensus_queue.cc:1035] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996a5b7ec8d94518898b6f0ad0a5cefc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:44.361857 16829 consensus_queue.cc:1035] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0df5dba621b4d7a9db0970fe012e855" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:44.373644 16836 mvcc.cc:205] Tried to move back new op lower bound from 7299113960889847808 to 7299113960569253888. Current Snapshot: MvccSnapshot[applied={T|T < 7299113960889847808}]
I20260621 02:50:44.376257 16835 mvcc.cc:205] Tried to move back new op lower bound from 7299113960889847808 to 7299113960569253888. Current Snapshot: MvccSnapshot[applied={T|T < 7299113960889847808 or (T in {7299113960889847808})}]
I20260621 02:50:44.385373 16356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:44.385583 16356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:44.388736 16356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64c38a4f021347b587b0d040728659d2: Sending DeleteTablet for 3 replicas of tablet 20e2efdd398c48a5be0e2cf74ec552ec
I20260621 02:50:44.389770 16738 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e2efdd398c48a5be0e2cf74ec552ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:44 UTC) from {username='slave'} at 127.0.0.1:44982
I20260621 02:50:44.389899 16608 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e2efdd398c48a5be0e2cf74ec552ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:44 UTC) from {username='slave'} at 127.0.0.1:50230
I20260621 02:50:44.390271 16478 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e2efdd398c48a5be0e2cf74ec552ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:44 UTC) from {username='slave'} at 127.0.0.1:52056
I20260621 02:50:44.392166 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16416
I20260621 02:50:44.400720 16846 tablet_replica.cc:330] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: stopping tablet replica
I20260621 02:50:44.401259 16846 raft_consensus.cc:2229] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:44.401816 16846 raft_consensus.cc:2258] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:44.407210 16845 tablet_replica.cc:330] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: stopping tablet replica
I20260621 02:50:44.408159 16845 raft_consensus.cc:2229] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:44.408870 16846 ts_tablet_manager.cc:1905] T 20e2efdd398c48a5be0e2cf74ec552ec P 996a5b7ec8d94518898b6f0ad0a5cefc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:44.409308 16845 raft_consensus.cc:2258] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:44.409358 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.393715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.11.109.1:35413)
0621 02:50:44.393914 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:44.393923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:44.407011 (+ 13088us) server_negotiation.cc:407] Connection header received
0621 02:50:44.407213 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:44.407219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:44.407299 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:44.407397 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:44.408522 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:44.408530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:44.408537 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:44.408555 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:44.408592 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:44.408606 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:44.408704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:44.408975 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 02:50:44.409033 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:44.410328 16538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:44.411095 16844 tablet_replica.cc:330] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: stopping tablet replica
I20260621 02:50:44.411530 16844 raft_consensus.cc:2229] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:44.412032 16844 raft_consensus.cc:2258] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:44.414105 16844 ts_tablet_manager.cc:1905] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:44.417388 16845 ts_tablet_manager.cc:1905] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:44.421376 16844 ts_tablet_manager.cc:1918] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:44.421617 16844 log.cc:1196] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/20e2efdd398c48a5be0e2cf74ec552ec
I20260621 02:50:44.422256 16844 ts_tablet_manager.cc:1939] T 20e2efdd398c48a5be0e2cf74ec552ec P d0df5dba621b4d7a9db0970fe012e855: Deleting consensus metadata
I20260621 02:50:44.424156 16342 catalog_manager.cc:4841] TS d0df5dba621b4d7a9db0970fe012e855 (127.11.109.3:41759): tablet 20e2efdd398c48a5be0e2cf74ec552ec (table test-table [id=ca59048083f24a918f9204ed133532b2]) successfully deleted
I20260621 02:50:44.433743 16845 ts_tablet_manager.cc:1918] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:44.433964 16845 log.cc:1196] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/20e2efdd398c48a5be0e2cf74ec552ec
I20260621 02:50:44.434520 16845 ts_tablet_manager.cc:1939] T 20e2efdd398c48a5be0e2cf74ec552ec P a0c736909d1b48059db08f8805a25a52: Deleting consensus metadata
I20260621 02:50:44.436157 16343 catalog_manager.cc:4841] TS a0c736909d1b48059db08f8805a25a52 (127.11.109.2:44349): tablet 20e2efdd398c48a5be0e2cf74ec552ec (table test-table [id=ca59048083f24a918f9204ed133532b2]) successfully deleted
I20260621 02:50:44.504160 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16416
I20260621 02:50:44.525857 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16546
W20260621 02:50:44.525893 16344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.109.1:35413 (error 108)
I20260621 02:50:44.528378 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.527588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52084)
0621 02:50:44.528031 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.528127 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
W20260621 02:50:44.528673 16344 catalog_manager.cc:4584] TS 996a5b7ec8d94518898b6f0ad0a5cefc (127.11.109.1:35413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20e2efdd398c48a5be0e2cf74ec552ec: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
I20260621 02:50:44.529502 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.526677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.11.109.2:44349)
0621 02:50:44.527235 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:44.527241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:44.527257 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:50:44.527338 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:44.527344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:44.527403 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:44.527509 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:44.528699 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:44.528706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:44.528711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:44.528730 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:44.528768 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:44.528790 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:44.528928 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:44.529166 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 02:50:44.529245 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260621 02:50:44.530208 16668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:44.582264 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.581801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52088)
0621 02:50:44.582027 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.582116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260621 02:50:44.583316 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.582975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52092)
0621 02:50:44.583137 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.583201 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 02:50:44.634644 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16546
I20260621 02:50:44.643527 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.643102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52100)
0621 02:50:44.643307 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.643398 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 02:50:44.644599 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.644180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52112)
0621 02:50:44.644336 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.644417 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 02:50:44.652753 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16676
I20260621 02:50:44.655506 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.653479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.11.109.3:41759)
0621 02:50:44.653656 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:44.653664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:44.653683 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:50:44.653896 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:44.653901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:44.654000 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:44.654121 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:44.654891 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:44.654896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:44.654901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:44.654917 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:44.654955 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:44.654966 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:44.655095 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:44.655184 (+ 89us) server_negotiation.cc:299] Negotiation successful
0621 02:50:44.655240 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:44.656105 16798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:44.725335 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.724783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52122)
0621 02:50:44.725032 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.725126 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 02:50:44.726383 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:44.726003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35413 (local address 127.0.0.1:52126)
0621 02:50:44.726206 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:44.726268 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260621 02:50:44.742250 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16676
I20260621 02:50:44.758283 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16326
I20260621 02:50:44.759377 16387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:44.861629 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16326
2026-06-21T02:50:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2805 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 02:50:44.902097 11700 mini_oidc.cc:152] Starting JWKS server
I20260621 02:50:44.904621 11700 webserver.cc:460] Webserver started at https://127.0.0.1:37443/ using document root <none> and password file <none>
I20260621 02:50:44.904891 11700 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:50:44.905488 11700 webserver.cc:460] Webserver started at http://127.0.0.1:43451/ using document root <none> and password file <none>
2026-06-21T02:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:44Z Disabled control of system clock
I20260621 02:50:44.929359 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41381
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:41381
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/testchainca.cert with env {}
W20260621 02:50:45.234211 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:45.234488 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:45.234647 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:45.241537 16859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:45.241643 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:45.241708 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:45.241806 16859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:45.241888 16859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:50:45.241981 16859 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:50:45.242033 16859 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37443/jwks.json
I20260621 02:50:45.251457 16859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:41381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:45.252736 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:45.254019 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:45.265555 16867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:45.265614 16865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:45.265626 16864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:45.266160 16859 server_base.cc:1034] running on GCE node
I20260621 02:50:45.266911 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:45.268189 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:45.269410 16859 hybrid_clock.cc:648] HybridClock initialized: now 1782010245269381 us; error 51 us; skew 500 ppm
I20260621 02:50:45.269752 16859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:45.271977 16859 webserver.cc:460] Webserver started at http://127.11.109.62:35225/ using document root <none> and password file <none>
I20260621 02:50:45.272632 16859 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:45.272787 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:45.273092 16859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:45.275831 16859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "cada489fe86c4a25b93019323e82fc62"
format_stamp: "Formatted at 2026-06-21 02:50:45 on dist-test-slave-bc52"
server_key: "c925987807d641caa10ce3d2d59e2c28"
server_key_iv: "58c170f392036ff5f1f86a88679afda0"
server_key_version: "encryptionkey@0"
I20260621 02:50:45.276584 16859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "cada489fe86c4a25b93019323e82fc62"
format_stamp: "Formatted at 2026-06-21 02:50:45 on dist-test-slave-bc52"
server_key: "c925987807d641caa10ce3d2d59e2c28"
server_key_iv: "58c170f392036ff5f1f86a88679afda0"
server_key_version: "encryptionkey@0"
I20260621 02:50:45.281935 16859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:50:45.285091 16873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:45.286405 16859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:50:45.286594 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "cada489fe86c4a25b93019323e82fc62"
format_stamp: "Formatted at 2026-06-21 02:50:45 on dist-test-slave-bc52"
server_key: "c925987807d641caa10ce3d2d59e2c28"
server_key_iv: "58c170f392036ff5f1f86a88679afda0"
server_key_version: "encryptionkey@0"
I20260621 02:50:45.286813 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:45.304394 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:45.305130 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:45.305454 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:45.322418 16859 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:41381
I20260621 02:50:45.322435 16924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:41381 every 8 connection(s)
I20260621 02:50:45.323649 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:45.328258 16925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:45.333873 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16859
I20260621 02:50:45.334103 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:45.334831 11700 external_mini_cluster.cc:1428] Setting key e30fb2522dfc6be08b26c9f8ffb40602
I20260621 02:50:45.338999 16925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62: Bootstrap starting.
I20260621 02:50:45.341985 16925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:45.342929 16925 log.cc:824] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:45.345558 16925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62: No bootstrap required, opened a new log
I20260621 02:50:45.347697 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:45.336663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.11.109.62:41381)
0621 02:50:45.337609 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:45.337620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:45.337657 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:50:45.339312 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:45.339334 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:45.339737 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:45.340101 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:45.340650 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:45.342002 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:45.343411 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:45.344068 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:45.344268 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:45.345486 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:45.345493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:45.345506 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:45.346078 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:45.346148 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:45.346168 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:45.346295 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:45.346520 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 02:50:45.346787 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":298,"threads_started":1}
I20260621 02:50:45.351598 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:50:45.353909 16925 raft_consensus.cc:348] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cada489fe86c4a25b93019323e82fc62" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41381 } }
I20260621 02:50:45.354283 16925 raft_consensus.cc:374] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:45.354404 16925 raft_consensus.cc:729] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cada489fe86c4a25b93019323e82fc62, State: Initialized, Role: FOLLOWER
I20260621 02:50:45.354971 16925 consensus_queue.cc:260] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cada489fe86c4a25b93019323e82fc62" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41381 } }
I20260621 02:50:45.355201 16925 raft_consensus.cc:388] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:45.355355 16925 raft_consensus.cc:482] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:45.355537 16925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:45.357654 16925 raft_consensus.cc:504] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cada489fe86c4a25b93019323e82fc62" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41381 } }
I20260621 02:50:45.358107 16925 leader_election.cc:302] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cada489fe86c4a25b93019323e82fc62; no voters:
I20260621 02:50:45.358660 16925 leader_election.cc:288] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:45.358808 16930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:45.359865 16930 raft_consensus.cc:686] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [term 1 LEADER]: Becoming Leader. State: Replica: cada489fe86c4a25b93019323e82fc62, State: Running, Role: LEADER
I20260621 02:50:45.360466 16930 consensus_queue.cc:237] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cada489fe86c4a25b93019323e82fc62" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41381 } }
I20260621 02:50:45.360723 16925 sys_catalog.cc:564] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:45.362957 16931 sys_catalog.cc:455] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cada489fe86c4a25b93019323e82fc62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cada489fe86c4a25b93019323e82fc62" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41381 } } }
I20260621 02:50:45.363286 16931 sys_catalog.cc:458] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:45.363791 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cada489fe86c4a25b93019323e82fc62. Latest consensus state: current_term: 1 leader_uuid: "cada489fe86c4a25b93019323e82fc62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cada489fe86c4a25b93019323e82fc62" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41381 } } }
I20260621 02:50:45.364068 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:45.365370 16937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:45.369313 16937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:45.379841 16937 catalog_manager.cc:1347] Generated new cluster ID: 7e60ae94c6f445fab73ce650f7cc47f2
I20260621 02:50:45.380018 16937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:45.399080 16937 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:45.400293 16937 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:45.409485 16937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cada489fe86c4a25b93019323e82fc62: Generated new TSK 0
I20260621 02:50:45.410239 16937 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 02:50:45.450114 16955 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 02:50:45.450282 16955 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:50:45.451483 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:45.430148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.11.109.62:41381)
0621 02:50:45.430355 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:45.430361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:45.430831 (+ 470us) server_negotiation.cc:407] Connection header received
0621 02:50:45.431087 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:45.431093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:45.431176 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:45.431329 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:50:45.431809 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:45.432952 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:45.434386 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:45.435107 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:45.435239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:45.435706 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:50:45.450824 (+ 15118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 02:50:45.451167 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:50:45.451615 16928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 02:50:45.452698 16954 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:45.430083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41381 (local address 127.0.0.1:40756)
0621 02:50:45.430735 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:45.430751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:45.430884 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:45.431352 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:45.431355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:45.431364 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:50:45.431671 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:45.431679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:45.433081 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:45.433085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:45.434256 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:45.434267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:45.435270 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:45.435274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:45.435388 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:45.435443 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:50:45.451061 (+ 15618us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 02:50:45.451308 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.109.62:41381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":336,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.109.62:41381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 02:50:45.454353 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16859
I20260621 02:50:45.455554 16920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:45.545689 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16859
2026-06-21T02:50:45Z chronyd exiting
I20260621 02:50:45.592585 11700 test_util.cc:175] -----------------------------------------------
I20260621 02:50:45.592729 11700 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782010204918124-11700-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 02:50:45.596316 11700 mini_oidc.cc:152] Starting JWKS server
I20260621 02:50:45.598160 11700 webserver.cc:460] Webserver started at https://127.0.0.1:36075/ using document root <none> and password file <none>
I20260621 02:50:45.598385 11700 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:50:45.599014 11700 webserver.cc:460] Webserver started at http://127.0.0.1:34059/ using document root <none> and password file <none>
2026-06-21T02:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:45Z Disabled control of system clock
I20260621 02:50:45.623211 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44401
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:36449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:44401
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36075/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/testchainca.cert with env {}
W20260621 02:50:45.931351 16962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:45.931597 16962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:45.931676 16962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:45.938546 16962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:45.938634 16962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:45.938673 16962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:45.938725 16962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:45.938760 16962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:50:45.938815 16962 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:50:45.938853 16962 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36075/jwks.json
I20260621 02:50:45.948218 16962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:44401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36075/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:45.949352 16962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:45.950543 16962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:45.961735 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:45.961810 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:45.961774 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:45.962574 16962 server_base.cc:1034] running on GCE node
I20260621 02:50:45.963312 16962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:45.964572 16962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:45.965795 16962 hybrid_clock.cc:648] HybridClock initialized: now 1782010245965760 us; error 52 us; skew 500 ppm
I20260621 02:50:45.966143 16962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:45.971100 16962 webserver.cc:460] Webserver started at http://127.11.109.62:43759/ using document root <none> and password file <none>
I20260621 02:50:45.971771 16962 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:45.971901 16962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:45.972226 16962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:45.974975 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "5fc47c8f4c56477e818121a28b9f7d2a"
format_stamp: "Formatted at 2026-06-21 02:50:45 on dist-test-slave-bc52"
server_key: "7278669ba1a0d9d059c483ea7b98481d"
server_key_iv: "896956da8ac7764a363514c8505eb733"
server_key_version: "encryptionkey@0"
I20260621 02:50:45.975755 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "5fc47c8f4c56477e818121a28b9f7d2a"
format_stamp: "Formatted at 2026-06-21 02:50:45 on dist-test-slave-bc52"
server_key: "7278669ba1a0d9d059c483ea7b98481d"
server_key_iv: "896956da8ac7764a363514c8505eb733"
server_key_version: "encryptionkey@0"
I20260621 02:50:45.980911 16962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:50:45.984113 16976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:45.985451 16962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:45.985616 16962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "5fc47c8f4c56477e818121a28b9f7d2a"
format_stamp: "Formatted at 2026-06-21 02:50:45 on dist-test-slave-bc52"
server_key: "7278669ba1a0d9d059c483ea7b98481d"
server_key_iv: "896956da8ac7764a363514c8505eb733"
server_key_version: "encryptionkey@0"
I20260621 02:50:45.985816 16962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:46.021863 16962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:46.022670 16962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:46.022975 16962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:46.040711 16962 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:44401
I20260621 02:50:46.040732 17027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:44401 every 8 connection(s)
I20260621 02:50:46.042030 16962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:46.045500 17028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:46.048450 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 16962
I20260621 02:50:46.048645 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:46.049444 11700 external_mini_cluster.cc:1428] Setting key 58524cb18b8af3fa73eea9c051b26237
I20260621 02:50:46.056978 17028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a: Bootstrap starting.
I20260621 02:50:46.059988 17028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:46.061085 17028 log.cc:824] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:46.062243 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:46.051299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.11.109.62:44401)
0621 02:50:46.052190 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:46.052200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:46.052238 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:50:46.053882 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:46.053901 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:46.054343 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:46.054720 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:46.055259 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.056462 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.057810 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.058477 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.058675 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:46.059783 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:46.059790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:46.059803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:46.060346 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:46.060425 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:46.060444 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:46.060570 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:46.060782 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 02:50:46.061073 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":326,"threads_started":1}
I20260621 02:50:46.064057 17028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a: No bootstrap required, opened a new log
I20260621 02:50:46.065752 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:50:46.071384 17028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44401 } }
I20260621 02:50:46.071719 17028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:46.071817 17028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc47c8f4c56477e818121a28b9f7d2a, State: Initialized, Role: FOLLOWER
I20260621 02:50:46.072482 17028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44401 } }
I20260621 02:50:46.072717 17028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:46.072854 17028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:46.073024 17028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:46.075239 17028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44401 } }
I20260621 02:50:46.075722 17028 leader_election.cc:302] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc47c8f4c56477e818121a28b9f7d2a; no voters:
I20260621 02:50:46.076243 17028 leader_election.cc:288] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:46.076359 17033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:46.077040 17033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [term 1 LEADER]: Becoming Leader. State: Replica: 5fc47c8f4c56477e818121a28b9f7d2a, State: Running, Role: LEADER
I20260621 02:50:46.077595 17033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44401 } }
I20260621 02:50:46.078209 17028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:46.080286 17034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44401 } } }
I20260621 02:50:46.080711 17034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:46.080574 17035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fc47c8f4c56477e818121a28b9f7d2a. Latest consensus state: current_term: 1 leader_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc47c8f4c56477e818121a28b9f7d2a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44401 } } }
I20260621 02:50:46.080864 17035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:46.082573 17039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:46.087703 17039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:46.095647 17039 catalog_manager.cc:1347] Generated new cluster ID: ddb38b5e64794f56abc7e5e86a07aa0d
I20260621 02:50:46.095747 17039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:46.115017 17039 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:46.116243 17039 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:46.122277 17039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fc47c8f4c56477e818121a28b9f7d2a: Generated new TSK 0
I20260621 02:50:46.123030 17039 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 02:50:46.159080 17058 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 02:50:46.159206 17058 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:50:46.160344 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:46.140828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.11.109.62:44401)
0621 02:50:46.141003 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:46.141010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:46.141385 (+ 375us) server_negotiation.cc:407] Connection header received
0621 02:50:46.141562 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:46.141568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:46.141659 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:46.141819 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:50:46.142269 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.143382 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.144761 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.145439 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.145565 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:46.145964 (+ 399us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:50:46.159718 (+ 13754us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 02:50:46.160070 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 02:50:46.160392 17057 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:46.140667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44401 (local address 127.0.0.1:39530)
0621 02:50:46.141292 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:46.141307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:46.141419 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:46.141854 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:46.141857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:46.141866 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:50:46.142115 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:46.142123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.143505 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.143509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:46.144621 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:46.144632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.145586 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.145590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:46.145711 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:46.145743 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:50:46.159925 (+ 14182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 02:50:46.160110 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.109.62:44401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":353,"threads_started":1}
W20260621 02:50:46.160459 17031 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260621 02:50:46.161687 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 16962
I20260621 02:50:46.162747 17023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:46.252337 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 16962
2026-06-21T02:50:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 02:50:46.306603 11700 mini_oidc.cc:152] Starting JWKS server
I20260621 02:50:46.308511 11700 webserver.cc:460] Webserver started at https://127.0.0.1:46653/ using document root <none> and password file <none>
I20260621 02:50:46.308744 11700 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:50:46.309391 11700 webserver.cc:460] Webserver started at http://127.0.0.1:35885/ using document root <none> and password file <none>
2026-06-21T02:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:46Z Disabled control of system clock
I20260621 02:50:46.335249 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:43425
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:43425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 02:50:46.645538 17065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:46.645902 17065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:46.646061 17065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:46.653393 17065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:46.653538 17065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:46.653633 17065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:46.653740 17065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:46.653825 17065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:50:46.653921 17065 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:50:46.653998 17065 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46653/jwks.json
I20260621 02:50:46.663044 17065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:43425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:46.664337 17065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:46.666112 17065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:46.677423 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:46.677464 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:46.677505 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:46.678153 17065 server_base.cc:1034] running on GCE node
I20260621 02:50:46.678869 17065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:46.680162 17065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:46.681423 17065 hybrid_clock.cc:648] HybridClock initialized: now 1782010246681368 us; error 58 us; skew 500 ppm
I20260621 02:50:46.681782 17065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:46.683986 17065 webserver.cc:460] Webserver started at http://127.11.109.62:32957/ using document root <none> and password file <none>
I20260621 02:50:46.684666 17065 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:46.684795 17065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:46.685119 17065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:46.687914 17065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "00b45c249fae448c85d387e815287b66"
format_stamp: "Formatted at 2026-06-21 02:50:46 on dist-test-slave-bc52"
server_key: "dffccc201223db8c269cc926df245c97"
server_key_iv: "6e9885c81c805fa9d5a69cfa48474139"
server_key_version: "encryptionkey@0"
I20260621 02:50:46.688701 17065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "00b45c249fae448c85d387e815287b66"
format_stamp: "Formatted at 2026-06-21 02:50:46 on dist-test-slave-bc52"
server_key: "dffccc201223db8c269cc926df245c97"
server_key_iv: "6e9885c81c805fa9d5a69cfa48474139"
server_key_version: "encryptionkey@0"
I20260621 02:50:46.694432 17065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260621 02:50:46.697808 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:46.699101 17065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:50:46.699277 17065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "00b45c249fae448c85d387e815287b66"
format_stamp: "Formatted at 2026-06-21 02:50:46 on dist-test-slave-bc52"
server_key: "dffccc201223db8c269cc926df245c97"
server_key_iv: "6e9885c81c805fa9d5a69cfa48474139"
server_key_version: "encryptionkey@0"
I20260621 02:50:46.699487 17065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:46.741612 17065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:46.742534 17065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:46.742842 17065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:46.761045 17065 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:43425
I20260621 02:50:46.761078 17130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:43425 every 8 connection(s)
I20260621 02:50:46.762384 17065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:46.762943 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17065
I20260621 02:50:46.763171 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:46.763967 11700 external_mini_cluster.cc:1428] Setting key f5d6e60a3809f1a60cb6e30cf50e76bd
I20260621 02:50:46.766628 17131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:46.777388 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:46.765860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.11.109.62:43425)
0621 02:50:46.766888 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:46.766900 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:46.766941 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:50:46.768447 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:46.768467 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:46.768902 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:46.769271 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:46.769795 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.771354 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.773254 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.773882 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.774001 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:46.775158 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:46.775165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:46.775179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:46.775727 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:46.775792 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:46.775813 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:46.775916 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:46.776145 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 02:50:46.776489 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":356,"threads_started":1}
I20260621 02:50:46.778227 17131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66: Bootstrap starting.
I20260621 02:50:46.781316 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:50:46.781845 17131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:46.782987 17131 log.cc:824] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:46.785830 17131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66: No bootstrap required, opened a new log
I20260621 02:50:46.792605 17131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b45c249fae448c85d387e815287b66" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 43425 } }
I20260621 02:50:46.792912 17131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:46.792982 17131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b45c249fae448c85d387e815287b66, State: Initialized, Role: FOLLOWER
I20260621 02:50:46.793494 17131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b45c249fae448c85d387e815287b66" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 43425 } }
I20260621 02:50:46.793732 17131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:46.793877 17131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:46.794049 17131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:46.796026 17131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b45c249fae448c85d387e815287b66" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 43425 } }
I20260621 02:50:46.796485 17131 leader_election.cc:302] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00b45c249fae448c85d387e815287b66; no voters:
I20260621 02:50:46.797080 17131 leader_election.cc:288] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:46.797235 17136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:46.798183 17136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [term 1 LEADER]: Becoming Leader. State: Replica: 00b45c249fae448c85d387e815287b66, State: Running, Role: LEADER
I20260621 02:50:46.798736 17136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b45c249fae448c85d387e815287b66" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 43425 } }
I20260621 02:50:46.799273 17131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:46.801523 17138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00b45c249fae448c85d387e815287b66. Latest consensus state: current_term: 1 leader_uuid: "00b45c249fae448c85d387e815287b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b45c249fae448c85d387e815287b66" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 43425 } } }
I20260621 02:50:46.801956 17138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:46.801566 17137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00b45c249fae448c85d387e815287b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b45c249fae448c85d387e815287b66" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 43425 } } }
I20260621 02:50:46.802670 17137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:46.804124 17145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:46.808336 17145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:46.815904 17145 catalog_manager.cc:1347] Generated new cluster ID: 4514fc9284714bdeb3401115f290c31d
I20260621 02:50:46.816001 17145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:46.827013 17145 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:46.828300 17145 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:46.841434 17145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00b45c249fae448c85d387e815287b66: Generated new TSK 0
I20260621 02:50:46.842193 17145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:50:46.865799 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:46.858525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.11.109.62:43425)
0621 02:50:46.858723 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:46.858732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:46.859112 (+ 380us) server_negotiation.cc:407] Connection header received
0621 02:50:46.859372 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:46.859376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:46.859445 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:46.859610 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:50:46.860057 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.860794 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.862249 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.862702 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.862832 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:46.863335 (+ 503us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:50:46.865109 (+ 1774us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 02:50:46.865500 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 02:50:46.866029 17160 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:46.858460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:43425 (local address 127.0.0.1:43804)
0621 02:50:46.859011 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:46.859027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:46.859186 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:46.859617 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:46.859622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:46.859634 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:50:46.859904 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:46.859913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.860971 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.860983 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:46.862098 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:46.862108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:46.862908 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:46.862916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:46.863055 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:46.863087 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:50:46.865356 (+ 2269us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 02:50:46.865594 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.109.62:43425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":322,"threads_started":1}
W20260621 02:50:46.866266 17134 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46653/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260621 02:50:46.869024 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17065
I20260621 02:50:46.870091 17126 generic_service.cc:196] Checking for leaks (request via RPC)
E20260621 02:50:46.878686 17161 webserver.cc:533] Webserver: sslize failed:
E20260621 02:50:46.878809 17161 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:50:46.960047 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17065
2026-06-21T02:50:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (707 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T02:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:47Z Disabled control of system clock
I20260621 02:50:47.038722 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:38473
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:38473
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/testchainca.cert with env {}
W20260621 02:50:47.349344 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:47.349628 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:47.349792 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:47.357100 17166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:47.357270 17166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:47.357311 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:47.357359 17166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:47.357445 17166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:50:47.357558 17166 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:50:47.357611 17166 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 02:50:47.367563 17166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:38473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:38473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:47.368860 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:47.370155 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:47.381682 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:47.381717 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:47.381639 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:47.382547 17166 server_base.cc:1034] running on GCE node
I20260621 02:50:47.383255 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:47.384570 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:47.385758 17166 hybrid_clock.cc:648] HybridClock initialized: now 1782010247385728 us; error 28 us; skew 500 ppm
I20260621 02:50:47.386099 17166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:50:47.391211 17166 webserver.cc:460] Webserver started at http://127.11.109.62:34369/ using document root <none> and password file <none>
I20260621 02:50:47.391849 17166 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:47.391976 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:47.392278 17166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:47.395112 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "dc5bfbfb1b06448d9524a201785e1079"
format_stamp: "Formatted at 2026-06-21 02:50:47 on dist-test-slave-bc52"
server_key: "ba62c2f7b24553e6312767f26d3283ed"
server_key_iv: "4850f300d21bc247bd21ea3124f9c761"
server_key_version: "encryptionkey@0"
I20260621 02:50:47.395874 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "dc5bfbfb1b06448d9524a201785e1079"
format_stamp: "Formatted at 2026-06-21 02:50:47 on dist-test-slave-bc52"
server_key: "ba62c2f7b24553e6312767f26d3283ed"
server_key_iv: "4850f300d21bc247bd21ea3124f9c761"
server_key_version: "encryptionkey@0"
I20260621 02:50:47.400544 17166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260621 02:50:47.403713 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:47.404954 17166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 02:50:47.405234 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "dc5bfbfb1b06448d9524a201785e1079"
format_stamp: "Formatted at 2026-06-21 02:50:47 on dist-test-slave-bc52"
server_key: "ba62c2f7b24553e6312767f26d3283ed"
server_key_iv: "4850f300d21bc247bd21ea3124f9c761"
server_key_version: "encryptionkey@0"
I20260621 02:50:47.405449 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:47.420629 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:47.421548 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:47.421762 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:47.439234 17166 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:38473
I20260621 02:50:47.439257 17231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:38473 every 8 connection(s)
I20260621 02:50:47.440490 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:47.444029 17232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:47.444698 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17166
I20260621 02:50:47.444893 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:47.445578 11700 external_mini_cluster.cc:1428] Setting key 9048e8dd986f79cc1b0d4dd84718a9c7
I20260621 02:50:47.458101 17232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079: Bootstrap starting.
I20260621 02:50:47.460052 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:47.447385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.11.109.62:38473)
0621 02:50:47.448422 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:47.448436 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:47.448486 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:50:47.450382 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:47.450409 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:47.450979 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:47.451395 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:47.451894 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:47.453748 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:47.455053 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:47.456108 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:47.456263 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:47.457372 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:47.457383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:47.457402 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:50:47.458099 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:47.458198 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:50:47.458222 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:47.458330 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:47.458542 (+ 212us) server_negotiation.cc:299] Negotiation successful
0621 02:50:47.458830 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":315,"threads_started":1}
I20260621 02:50:47.461808 17232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:47.462965 17232 log.cc:824] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:47.463768 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:50:47.465617 17232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079: No bootstrap required, opened a new log
I20260621 02:50:47.472357 17232 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5bfbfb1b06448d9524a201785e1079" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38473 } }
I20260621 02:50:47.472694 17232 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:47.472800 17232 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5bfbfb1b06448d9524a201785e1079, State: Initialized, Role: FOLLOWER
I20260621 02:50:47.473352 17232 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5bfbfb1b06448d9524a201785e1079" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38473 } }
I20260621 02:50:47.473583 17232 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:47.473719 17232 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:47.473882 17232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:47.475943 17232 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5bfbfb1b06448d9524a201785e1079" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38473 } }
I20260621 02:50:47.476409 17232 leader_election.cc:302] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc5bfbfb1b06448d9524a201785e1079; no voters:
I20260621 02:50:47.476967 17232 leader_election.cc:288] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:47.477099 17237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:47.477846 17237 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [term 1 LEADER]: Becoming Leader. State: Replica: dc5bfbfb1b06448d9524a201785e1079, State: Running, Role: LEADER
I20260621 02:50:47.478433 17237 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5bfbfb1b06448d9524a201785e1079" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38473 } }
I20260621 02:50:47.478951 17232 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:47.481451 17239 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5bfbfb1b06448d9524a201785e1079. Latest consensus state: current_term: 1 leader_uuid: "dc5bfbfb1b06448d9524a201785e1079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5bfbfb1b06448d9524a201785e1079" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38473 } } }
I20260621 02:50:47.481564 17238 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc5bfbfb1b06448d9524a201785e1079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5bfbfb1b06448d9524a201785e1079" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38473 } } }
I20260621 02:50:47.481911 17239 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:47.482029 17238 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:47.483637 17246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:47.488533 17246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:47.496438 17246 catalog_manager.cc:1347] Generated new cluster ID: 4b0e09ed89464db6b340d631c8714a01
I20260621 02:50:47.496533 17246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:47.520223 17246 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:47.522033 17246 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:47.550631 17246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc5bfbfb1b06448d9524a201785e1079: Generated new TSK 0
I20260621 02:50:47.551713 17246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:50:47.624609 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:47.585466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.11.109.62:38473)
0621 02:50:47.585696 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:47.585706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:47.586238 (+ 532us) server_negotiation.cc:407] Connection header received
0621 02:50:47.586533 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:47.586540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:47.586649 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:47.586819 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:50:47.587287 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:47.588159 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:47.589926 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:47.590354 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:47.590488 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:47.591007 (+ 519us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:50:47.623878 (+ 32871us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 02:50:47.624282 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56348: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 02:50:47.624756 17235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56348: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 02:50:47.624864 17261 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:47.585462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:38473 (local address 127.0.0.1:56348)
0621 02:50:47.586126 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:47.586144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:47.586312 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:47.586843 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:47.586848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:47.586862 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:50:47.587135 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:47.587145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:47.588299 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:47.588305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:47.589740 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:47.589754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:47.590520 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:47.590525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:47.590697 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:47.590760 (+ 63us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:50:47.624167 (+ 33407us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 02:50:47.624429 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.109.62:38473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":385,"threads_started":1}
I20260621 02:50:47.626431 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17166
I20260621 02:50:47.627498 17227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:47.722574 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17166
2026-06-21T02:50:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (742 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (394 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:48 dist-test-slave-bc52 krb5kdc[17276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:48 dist-test-slave-bc52 krb5kdc[17276](info): set up 2 sockets
Jun 21 02:50:48 dist-test-slave-bc52 krb5kdc[17276](info): commencing operation
krb5kdc: starting...
W20260621 02:50:49.589766 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:49 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:50:49.780113 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:36493
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:50.094435 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:50.094708 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:50.094794 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:50.101936 17289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:50.102041 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:50.102113 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:50.102212 17289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:50.102290 17289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:50.111891 17289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:50.113193 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:50.114461 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:50.126272 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:50.126188 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:50.126232 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:50.127364 17289 server_base.cc:1034] running on GCE node
I20260621 02:50:50.128152 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:50.129594 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:50.130820 17289 hybrid_clock.cc:648] HybridClock initialized: now 1782010250130777 us; error 52 us; skew 500 ppm
I20260621 02:50:50.131160 17289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:50 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010250, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:50.134233 17289 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:50:50.135664 17289 webserver.cc:460] Webserver started at http://127.11.109.62:40263/ using document root <none> and password file <none>
I20260621 02:50:50.136339 17289 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:50.136492 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:50.136801 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:50.139709 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "805f42cbc7f5497e970801581594077a"
format_stamp: "Formatted at 2026-06-21 02:50:50 on dist-test-slave-bc52"
server_key: "a2508d89ce31c7fd759353c4ade66de2"
server_key_iv: "3adc34c6c3c1470dad40cd4fe285ce40"
server_key_version: "encryptionkey@0"
I20260621 02:50:50.140515 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "805f42cbc7f5497e970801581594077a"
format_stamp: "Formatted at 2026-06-21 02:50:50 on dist-test-slave-bc52"
server_key: "a2508d89ce31c7fd759353c4ade66de2"
server_key_iv: "3adc34c6c3c1470dad40cd4fe285ce40"
server_key_version: "encryptionkey@0"
I20260621 02:50:50.145833 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 02:50:50.149150 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:50.150501 17289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:50:50.150677 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "805f42cbc7f5497e970801581594077a"
format_stamp: "Formatted at 2026-06-21 02:50:50 on dist-test-slave-bc52"
server_key: "a2508d89ce31c7fd759353c4ade66de2"
server_key_iv: "3adc34c6c3c1470dad40cd4fe285ce40"
server_key_version: "encryptionkey@0"
I20260621 02:50:50.150869 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:50.164808 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:50.169037 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:50.169377 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:50.187536 17289 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:36493
I20260621 02:50:50.187574 17355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:36493 every 8 connection(s)
I20260621 02:50:50.188862 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:50.190433 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17289
I20260621 02:50:50.190649 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:50.191280 11700 external_mini_cluster.cc:1428] Setting key 887aa7a3e41bedd75fb979ee87cc47c8
I20260621 02:50:50.192672 17356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:50:50 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:50.206067 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a: Bootstrap starting.
I20260621 02:50:50.209960 17356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:50.211149 17356 log.cc:824] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:50.214277 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a: No bootstrap required, opened a new log
I20260621 02:50:50.214586 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:50.193243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.11.109.62:36493)
0621 02:50:50.194031 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:50.194046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:50.194105 (+ 59us) server_negotiation.cc:407] Connection header received
0621 02:50:50.195947 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:50.195973 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:50.196464 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:50.196889 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:50.198135 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:50.200106 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:50.201321 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:50.202301 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:50.202493 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:50.206637 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:50.206647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:50.206663 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:50.206701 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:50.210566 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:50.211187 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:50.211195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:50.211202 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:50.211388 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:50.211809 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:50.211820 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:50.211825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:50.212336 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:50.212616 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:50.212958 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 02:50:50.213361 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260621 02:50:50.221771 17356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805f42cbc7f5497e970801581594077a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36493 } }
I20260621 02:50:50.222069 17356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:50.222142 17356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805f42cbc7f5497e970801581594077a, State: Initialized, Role: FOLLOWER
I20260621 02:50:50.222649 17356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805f42cbc7f5497e970801581594077a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36493 } }
I20260621 02:50:50.222884 17356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:50.223042 17356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:50.223223 17356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:50.225234 17356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805f42cbc7f5497e970801581594077a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36493 } }
I20260621 02:50:50.225699 17356 leader_election.cc:302] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 805f42cbc7f5497e970801581594077a; no voters:
I20260621 02:50:50.226218 17356 leader_election.cc:288] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:50.226341 17361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:50.227694 17361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [term 1 LEADER]: Becoming Leader. State: Replica: 805f42cbc7f5497e970801581594077a, State: Running, Role: LEADER
I20260621 02:50:50.228229 17356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:50.228248 17361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805f42cbc7f5497e970801581594077a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36493 } }
I20260621 02:50:50.231066 17363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805f42cbc7f5497e970801581594077a. Latest consensus state: current_term: 1 leader_uuid: "805f42cbc7f5497e970801581594077a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805f42cbc7f5497e970801581594077a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36493 } } }
I20260621 02:50:50.231452 17363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:50.231688 17362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "805f42cbc7f5497e970801581594077a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805f42cbc7f5497e970801581594077a" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36493 } } }
I20260621 02:50:50.232052 17362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:50.233448 17369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:50.238658 17369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:50.246968 17369 catalog_manager.cc:1347] Generated new cluster ID: 91e8780090e845e98f99bfc09f9200d9
I20260621 02:50:50.247063 17369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:50.273653 17369 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:50.275463 17369 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:50.282960 17369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 805f42cbc7f5497e970801581594077a: Generated new TSK 0
I20260621 02:50:50.283891 17369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:50:50.334499 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36493
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:50.647850 17382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:50.648103 17382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:50.648193 17382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:50.655452 17382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:50.655541 17382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:50.655723 17382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:50.664851 17382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:50.666004 17382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:50.667164 17382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:50.679286 17388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:50.679498 17387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:50.679519 17390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:50.680032 17382 server_base.cc:1034] running on GCE node
I20260621 02:50:50.680552 17382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:50.681427 17382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:50.682617 17382 hybrid_clock.cc:648] HybridClock initialized: now 1782010250682565 us; error 47 us; skew 500 ppm
I20260621 02:50:50.682843 17382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:50 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010250, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:50.685861 17382 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:50:50.687148 17382 webserver.cc:460] Webserver started at http://127.11.109.1:41577/ using document root <none> and password file <none>
I20260621 02:50:50.687768 17382 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:50.687923 17382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:50.688206 17382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:50.690908 17382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "7df3a99043e848db8ee424080f143f96"
format_stamp: "Formatted at 2026-06-21 02:50:50 on dist-test-slave-bc52"
server_key: "701ad7ac456af26901aaa2ba640350fe"
server_key_iv: "37cda6893315c86203fa8b1497059161"
server_key_version: "encryptionkey@0"
I20260621 02:50:50.691684 17382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "7df3a99043e848db8ee424080f143f96"
format_stamp: "Formatted at 2026-06-21 02:50:50 on dist-test-slave-bc52"
server_key: "701ad7ac456af26901aaa2ba640350fe"
server_key_iv: "37cda6893315c86203fa8b1497059161"
server_key_version: "encryptionkey@0"
I20260621 02:50:50.696476 17382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 02:50:50.699898 17397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:50.701282 17382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:50.701468 17382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "7df3a99043e848db8ee424080f143f96"
format_stamp: "Formatted at 2026-06-21 02:50:50 on dist-test-slave-bc52"
server_key: "701ad7ac456af26901aaa2ba640350fe"
server_key_iv: "37cda6893315c86203fa8b1497059161"
server_key_version: "encryptionkey@0"
I20260621 02:50:50.701665 17382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:50.715031 17382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:50.719180 17382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:50.719478 17382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:50.720337 17382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:50.721705 17382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:50.721807 17382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:50.721956 17382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:50.722033 17382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:50.754259 17382 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:43439
I20260621 02:50:50.754276 17509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:43439 every 8 connection(s)
I20260621 02:50:50.755664 17382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:50.762271 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17382
I20260621 02:50:50.762452 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:50.763077 11700 external_mini_cluster.cc:1428] Setting key 5a30fd866f40d8432b8088904e297ad4
Jun 21 02:50:50 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010250, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:50.775218 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:50.758223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:54473 (local address 127.11.109.62:36493)
0621 02:50:50.758678 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:50.758683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:50.759725 (+ 1042us) server_negotiation.cc:407] Connection header received
0621 02:50:50.761643 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:50.761649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:50.761736 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:50.761856 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:50.763463 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:50.764154 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:50.765670 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:50.766065 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:50.766200 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:50.769570 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:50.769575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:50.769579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:50.769608 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:50.772082 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:50.772990 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:50.772994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:50.772996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:50.773117 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:50.773613 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:50.773616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:50.773618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:50.773877 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:50.773984 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:50.774812 (+ 828us) server_negotiation.cc:299] Negotiation successful
0621 02:50:50.774959 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":249,"threads_started":1}
I20260621 02:50:50.776767 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:50.758819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36493 (local address 127.11.109.1:54473)
0621 02:50:50.759515 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:50.759552 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:50.761385 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:50.762087 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:50.762095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:50.762701 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:50.763293 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:50.763312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:50.764307 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:50.764312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:50.765524 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:50.765535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:50.766282 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:50.766291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:50.766526 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:50.767200 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:50.767221 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:50.769361 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:50.772220 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:50.772231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:50.772249 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:50.772855 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:50.773254 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:50.773261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:50.773267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:50.773502 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:50.774001 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:50.774011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:50.774444 (+ 433us) client_negotiation.cc:769] Sending connection context
0621 02:50:50.774737 (+ 293us) client_negotiation.cc:240] Negotiation successful
0621 02:50:50.775283 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260621 02:50:50.778383 17510 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36493
I20260621 02:50:50.778692 17510 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:50.779561 17510 heartbeater.cc:511] Master 127.11.109.62:36493 requested a full tablet report, sending...
I20260621 02:50:50.781675 17321 ts_manager.cc:194] Registered new tserver with Master: 7df3a99043e848db8ee424080f143f96 (127.11.109.1:43439)
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:50:50.783479 17321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:54473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:50:50.806634 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36493
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:51.114784 17516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:51.115054 17516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:51.115145 17516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:51.121906 17516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:51.122004 17516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:51.122180 17516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:51.131400 17516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:51.132613 17516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:51.133919 17516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:51.146219 17524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:51.146322 17522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:51.146517 17521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:51.147126 17516 server_base.cc:1034] running on GCE node
I20260621 02:50:51.147714 17516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:51.148528 17516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:51.149730 17516 hybrid_clock.cc:648] HybridClock initialized: now 1782010251149698 us; error 35 us; skew 500 ppm
I20260621 02:50:51.150174 17516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.156152 17516 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:50:51.157588 17516 webserver.cc:460] Webserver started at http://127.11.109.2:44785/ using document root <none> and password file <none>
I20260621 02:50:51.158210 17516 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:51.158347 17516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:51.158636 17516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:51.161515 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "540472acba0d4e6f9fbfa134cb96b16d"
format_stamp: "Formatted at 2026-06-21 02:50:51 on dist-test-slave-bc52"
server_key: "2e30a7a4dd12409d819213da3556648a"
server_key_iv: "0dc705b59fbb8f877b796b0e62b05c04"
server_key_version: "encryptionkey@0"
I20260621 02:50:51.162371 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "540472acba0d4e6f9fbfa134cb96b16d"
format_stamp: "Formatted at 2026-06-21 02:50:51 on dist-test-slave-bc52"
server_key: "2e30a7a4dd12409d819213da3556648a"
server_key_iv: "0dc705b59fbb8f877b796b0e62b05c04"
server_key_version: "encryptionkey@0"
I20260621 02:50:51.167116 17516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:50:51.170181 17531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:51.171375 17516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:50:51.171550 17516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "540472acba0d4e6f9fbfa134cb96b16d"
format_stamp: "Formatted at 2026-06-21 02:50:51 on dist-test-slave-bc52"
server_key: "2e30a7a4dd12409d819213da3556648a"
server_key_iv: "0dc705b59fbb8f877b796b0e62b05c04"
server_key_version: "encryptionkey@0"
I20260621 02:50:51.171736 17516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:51.189983 17516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:51.194061 17516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:51.194332 17516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:51.195233 17516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:51.196496 17516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:51.196594 17516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:51.196736 17516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:51.196807 17516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:51.228114 17516 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:41647
I20260621 02:50:51.228196 17643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:41647 every 8 connection(s)
I20260621 02:50:51.229492 17516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:51.233366 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17516
I20260621 02:50:51.233551 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:51.234035 11700 external_mini_cluster.cc:1428] Setting key 041a8d8ef7386ab7abb839f01f7c4ea0
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:51.249213 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.231853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:44169 (local address 127.11.109.62:36493)
0621 02:50:51.232021 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.232026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.233291 (+ 1265us) server_negotiation.cc:407] Connection header received
0621 02:50:51.235315 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.235321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.235395 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.235501 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.236960 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.238239 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.239796 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.240538 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.240675 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.243849 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.243854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.243857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.243887 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.246342 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.247041 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.247053 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.247056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.247193 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.247635 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.247640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.247643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.247887 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.248040 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.248660 (+ 620us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.248884 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:50:51.250067 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.232349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36493 (local address 127.11.109.2:44169)
0621 02:50:51.233078 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:51.233117 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:51.235053 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:51.235723 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:51.235735 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:51.236254 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:51.236786 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:51.236802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.238365 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.238369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:51.239658 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:51.239669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.240723 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.240728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:51.240915 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.241617 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:51.241641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:51.243643 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:51.246478 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:51.246487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:51.246500 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:51.246900 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:51.247300 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:51.247305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:51.247309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:51.247476 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:51.248010 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:51.248017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:51.248321 (+ 304us) client_negotiation.cc:769] Sending connection context
0621 02:50:51.248570 (+ 249us) client_negotiation.cc:240] Negotiation successful
0621 02:50:51.249040 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":247,"threads_started":1}
I20260621 02:50:51.251436 17644 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36493
I20260621 02:50:51.251718 17644 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:51.252533 17644 heartbeater.cc:511] Master 127.11.109.62:36493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:50:51.254067 17321 ts_manager.cc:194] Registered new tserver with Master: 540472acba0d4e6f9fbfa134cb96b16d (127.11.109.2:41647)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
I20260621 02:50:51.255076 17321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:44169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:50:51.276463 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36493
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:51.585068 17649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:51.585359 17649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:51.585448 17649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:51.592280 17649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:51.592368 17649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:51.592478 17649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:51.601289 17649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:51.602470 17649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:51.603650 17649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:51.616222 17654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:51.616273 17655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:51.616312 17657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:51.617075 17649 server_base.cc:1034] running on GCE node
I20260621 02:50:51.617662 17649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:51.618448 17649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:51.619665 17649 hybrid_clock.cc:648] HybridClock initialized: now 1782010251619632 us; error 46 us; skew 500 ppm
I20260621 02:50:51.619953 17649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.622982 17649 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:50:51.624399 17649 webserver.cc:460] Webserver started at http://127.11.109.3:45401/ using document root <none> and password file <none>
I20260621 02:50:51.625089 17649 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:51.625255 17649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:51.625561 17649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:51.628402 17649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "48cbbf24cb7e41d2af0a44aa74c053b1"
format_stamp: "Formatted at 2026-06-21 02:50:51 on dist-test-slave-bc52"
server_key: "8646979a0d8823a0899d59c60e7961a9"
server_key_iv: "b3cf6958d4363851b0709f8b4bf19e75"
server_key_version: "encryptionkey@0"
I20260621 02:50:51.629267 17649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "48cbbf24cb7e41d2af0a44aa74c053b1"
format_stamp: "Formatted at 2026-06-21 02:50:51 on dist-test-slave-bc52"
server_key: "8646979a0d8823a0899d59c60e7961a9"
server_key_iv: "b3cf6958d4363851b0709f8b4bf19e75"
server_key_version: "encryptionkey@0"
I20260621 02:50:51.634079 17649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260621 02:50:51.637125 17664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:51.638334 17649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260621 02:50:51.638511 17649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "48cbbf24cb7e41d2af0a44aa74c053b1"
format_stamp: "Formatted at 2026-06-21 02:50:51 on dist-test-slave-bc52"
server_key: "8646979a0d8823a0899d59c60e7961a9"
server_key_iv: "b3cf6958d4363851b0709f8b4bf19e75"
server_key_version: "encryptionkey@0"
I20260621 02:50:51.638695 17649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:51.654585 17649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:51.658682 17649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:51.658982 17649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:51.659852 17649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:51.661149 17649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:51.661288 17649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 02:50:51.661432 17649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:51.661504 17649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:51.693898 17649 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:41085
I20260621 02:50:51.693933 17776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:41085 every 8 connection(s)
I20260621 02:50:51.695142 17649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:51.703622 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17649
I20260621 02:50:51.703817 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:51.704304 11700 external_mini_cluster.cc:1428] Setting key ac6cbdb027a2098aa3b773ec24534b83
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:50:51.714625 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.697634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:38241 (local address 127.11.109.62:36493)
0621 02:50:51.697804 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.697809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.699004 (+ 1195us) server_negotiation.cc:407] Connection header received
0621 02:50:51.700815 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.700820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.700929 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.701044 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.702547 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.703288 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.704999 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.705747 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.705870 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.709371 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.709377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.709383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.709424 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.711694 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.712360 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.712367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.712371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.712494 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.712902 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.712940 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.712943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.713296 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.713447 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.714082 (+ 635us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.714283 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:50:51.715634 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.698054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36493 (local address 127.11.109.3:38241)
0621 02:50:51.698797 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:51.698841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:51.700536 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:51.701298 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:51.701306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:51.701826 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:51.702368 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:51.702385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.703457 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.703461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:51.704846 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:51.704858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.705966 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.705977 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:51.706306 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.707018 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:51.707051 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:51.709128 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:51.711831 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:51.711839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:51.711854 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:51.712243 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:51.712604 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:51.712608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:51.712612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:51.712794 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:51.713441 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:51.713448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:51.713757 (+ 309us) client_negotiation.cc:769] Sending connection context
0621 02:50:51.714014 (+ 257us) client_negotiation.cc:240] Negotiation successful
0621 02:50:51.714513 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":308,"threads_started":1}
I20260621 02:50:51.716966 17777 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36493
I20260621 02:50:51.717296 17777 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:51.718084 17777 heartbeater.cc:511] Master 127.11.109.62:36493 requested a full tablet report, sending...
I20260621 02:50:51.719592 17321 ts_manager.cc:194] Registered new tserver with Master: 48cbbf24cb7e41d2af0a44aa74c053b1 (127.11.109.3:41085)
I20260621 02:50:51.720359 17321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:38241
I20260621 02:50:51.731478 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/test-admin.keytab.
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.758910 11700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 02:50:51.763859 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:51.761329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.11.109.1:43439)
0621 02:50:51.761910 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.761920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.761951 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:50:51.762172 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.762181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.762374 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.762543 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.763425 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59468: BlockingRecv error: recv got EOF from 127.0.0.1:59468 (error 108)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":289,"threads_started":1}
W20260621 02:50:51.764009 11700 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.765554 11700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.766798 11700 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:51.781627 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.768226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.11.109.1:43439)
0621 02:50:51.768417 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.768422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.768607 (+ 185us) server_negotiation.cc:407] Connection header received
0621 02:50:51.768733 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.768736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.768786 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.768866 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.769763 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.770704 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.771803 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.772323 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.772419 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.775780 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.775790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.775803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.775831 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.778464 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.779048 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.779055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.779057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.779165 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.779607 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.779612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.779616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.780098 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.780413 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.780832 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.781160 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:50:51.786454 17510 heartbeater.cc:503] Master 127.11.109.62:36493 was elected leader, sending a full tablet report...
W20260621 02:50:51.789130 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:51.786887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.11.109.1:43439)
0621 02:50:51.787408 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.787413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.787480 (+ 67us) server_negotiation.cc:407] Connection header received
0621 02:50:51.787533 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.787565 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.787673 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.787809 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.788902 (+ 1093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59486: BlockingRecv error: recv got EOF from 127.0.0.1:59486 (error 108)
Metrics: {"server-negotiator.queue_time_us":364}
W20260621 02:50:51.789752 11700 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.791114 11700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.792316 11700 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:51.805217 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.793649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.11.109.1:43439)
0621 02:50:51.793795 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.793799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.794013 (+ 214us) server_negotiation.cc:407] Connection header received
0621 02:50:51.794094 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.794097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.794143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.794216 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.795036 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.795703 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.796885 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.797474 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.797576 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.800854 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.800859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.800862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.800889 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.803147 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.803690 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.803694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.803696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.803803 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.804103 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.804106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.804108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.804362 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.804629 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.804761 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.804967 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260621 02:50:51.808988 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:51.807562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.11.109.1:43439)
0621 02:50:51.807768 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.807773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.807942 (+ 169us) server_negotiation.cc:407] Connection header received
0621 02:50:51.808047 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.808050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.808113 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.808184 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.808765 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59514: BlockingRecv error: recv got EOF from 127.0.0.1:59514 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:50:51.809265 11700 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782010204918124-11700-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.810650 11700 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:51.811820 11700 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:50:51.825076 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.812894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.11.109.1:43439)
0621 02:50:51.813093 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.813101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.813429 (+ 328us) server_negotiation.cc:407] Connection header received
0621 02:50:51.813586 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.813591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.813648 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.813743 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.814643 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.815321 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.816491 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.816903 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.817025 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.820301 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.820314 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.820318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.820345 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.822698 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.823310 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.823321 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.823325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.823443 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.823799 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.823806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.823809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.824054 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.824353 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.824603 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.824794 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:50:51.826514 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17382
I20260621 02:50:51.837131 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.827179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.11.109.1:43439)
0621 02:50:51.827309 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.827312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.827554 (+ 242us) server_negotiation.cc:407] Connection header received
0621 02:50:51.827665 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.827668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.827713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.827786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.828665 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.829376 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.830557 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.830909 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.831002 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.832809 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.832813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.832816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.832841 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.835145 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.835696 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.835699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.835701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.835810 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.836101 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.836104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.836105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.836309 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.836502 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.836758 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.836892 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:50:51.837742 17505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:51.918390 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17382
I20260621 02:50:51.935669 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17516
Jun 21 02:50:51 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:50:51.950409 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:51.936533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.11.109.2:41647)
0621 02:50:51.937116 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:51.937123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:51.937148 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:50:51.937263 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:51.937269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:51.937488 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:51.937666 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:51.938528 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.939488 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.940636 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:51.941314 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:51.941415 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:51.945248 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:51.945258 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:51.945271 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:51.945300 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:51.947901 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.948502 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.948508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.948511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.948617 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:51.948899 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:51.948901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:51.948903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:51.949299 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:51.949556 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:51.949843 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:50:51.950075 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260621 02:50:51.951635 17639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:52.032784 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17516
I20260621 02:50:52.048373 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17649
Jun 21 02:50:52 dist-test-slave-bc52 krb5kdc[17276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:50:52.063381 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:52.049290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.11.109.3:41085)
0621 02:50:52.049857 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:52.049864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:52.049891 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:50:52.049948 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:52.049954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:52.050167 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:52.050337 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:52.051409 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:52.052402 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:52.053804 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:52.054416 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:52.054561 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:52.058495 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:52.058505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:52.058519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:52.058551 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:52.061138 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:52.061803 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:52.061810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:52.061812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:52.061923 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:52.062218 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:52.062221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:52.062222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:52.062525 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:52.062764 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:52.062926 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 02:50:52.063105 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":280,"threads_started":1}
I20260621 02:50:52.064601 17772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:52.147208 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17649
I20260621 02:50:52.163131 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17289
I20260621 02:50:52.164213 17351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:52.266547 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17289
2026-06-21T02:50:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3755 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:50:52 dist-test-slave-bc52 krb5kdc[17804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:50:52 dist-test-slave-bc52 krb5kdc[17804](info): set up 2 sockets
Jun 21 02:50:52 dist-test-slave-bc52 krb5kdc[17804](info): commencing operation
krb5kdc: starting...
W20260621 02:50:53.349579 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:50:53 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:50:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab.
Entry for principal oryx/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab.
I20260621 02:50:53.537215 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:36053
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab
--principal=oryx/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:53.843786 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:53.844089 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:53.844242 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:53.851164 17817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:53.851291 17817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:53.851382 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:53.851475 17817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:53.851552 17817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:53.860884 17817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:36053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab
--principal=oryx/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:53.862136 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:53.863435 17817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:53.874491 17823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:53.874524 17822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:53.874969 17825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:53.875254 17817 server_base.cc:1034] running on GCE node
I20260621 02:50:53.875950 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:53.877298 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:53.878491 17817 hybrid_clock.cc:648] HybridClock initialized: now 1782010253878452 us; error 30 us; skew 500 ppm
I20260621 02:50:53.878818 17817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:53 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:53.881943 17817 init.cc:377] Logged in from keytab as oryx/127.11.109.62@KRBTEST.COM (short username oryx)
I20260621 02:50:53.883412 17817 webserver.cc:460] Webserver started at http://127.11.109.62:34367/ using document root <none> and password file <none>
I20260621 02:50:53.884049 17817 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:53.884184 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:53.884470 17817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:53.887267 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "b547e792b86941398e04e3f6fb122765"
format_stamp: "Formatted at 2026-06-21 02:50:53 on dist-test-slave-bc52"
server_key: "e749ff5bc74e3e5a4cd9f39998bac392"
server_key_iv: "f7526dd17c9ae53b398c7c2b4a16ab60"
server_key_version: "encryptionkey@0"
I20260621 02:50:53.888003 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "b547e792b86941398e04e3f6fb122765"
format_stamp: "Formatted at 2026-06-21 02:50:53 on dist-test-slave-bc52"
server_key: "e749ff5bc74e3e5a4cd9f39998bac392"
server_key_iv: "f7526dd17c9ae53b398c7c2b4a16ab60"
server_key_version: "encryptionkey@0"
I20260621 02:50:53.892673 17817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:50:53.895891 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:53.897229 17817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:50:53.897390 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "b547e792b86941398e04e3f6fb122765"
format_stamp: "Formatted at 2026-06-21 02:50:53 on dist-test-slave-bc52"
server_key: "e749ff5bc74e3e5a4cd9f39998bac392"
server_key_iv: "f7526dd17c9ae53b398c7c2b4a16ab60"
server_key_version: "encryptionkey@0"
I20260621 02:50:53.897570 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:53.911803 17817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:53.915521 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:53.915783 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:53.933347 17817 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:36053
I20260621 02:50:53.933353 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:36053 every 8 connection(s)
I20260621 02:50:53.934655 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:50:53.938154 17884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:53.944101 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17817
I20260621 02:50:53.944293 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:50:53.944916 11700 external_mini_cluster.cc:1428] Setting key cd63d571ed64147066f3d9b3b290e9b8
I20260621 02:50:53.948057 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Bootstrap starting.
I20260621 02:50:53.951371 17884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:53.952472 17884 log.cc:824] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:53.955338 17884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: No bootstrap required, opened a new log
Jun 21 02:50:53 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:53.964624 17884 raft_consensus.cc:348] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } }
I20260621 02:50:53.965060 17884 raft_consensus.cc:374] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:53.965168 17884 raft_consensus.cc:729] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b547e792b86941398e04e3f6fb122765, State: Initialized, Role: FOLLOWER
I20260621 02:50:53.965874 17884 consensus_queue.cc:260] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } }
I20260621 02:50:53.966188 17884 raft_consensus.cc:388] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:50:53.966327 17884 raft_consensus.cc:482] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:50:53.966511 17884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:53.967809 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:53.946792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.11.109.62:36053)
0621 02:50:53.947583 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:53.947594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:53.947632 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:50:53.949351 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:53.949371 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:53.949732 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:53.950029 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:53.951770 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:53.953072 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:53.954483 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:53.955537 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:53.955744 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:53.959391 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:53.959404 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:53.959423 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:53.959479 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:53.963412 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:53.964028 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:53.964036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:53.964043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:53.964229 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:53.964550 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:53.964555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:53.964558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:53.965267 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:53.965572 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:53.966011 (+ 439us) server_negotiation.cc:299] Negotiation successful
0621 02:50:53.966428 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":353,"threads_started":1}
I20260621 02:50:53.969699 17884 raft_consensus.cc:504] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } }
I20260621 02:50:53.970168 17884 leader_election.cc:302] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b547e792b86941398e04e3f6fb122765; no voters:
I20260621 02:50:53.970849 17884 leader_election.cc:288] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:50:53.970973 17889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:53.971664 17889 raft_consensus.cc:686] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 LEADER]: Becoming Leader. State: Replica: b547e792b86941398e04e3f6fb122765, State: Running, Role: LEADER
I20260621 02:50:53.972177 17889 consensus_queue.cc:237] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } }
I20260621 02:50:53.972785 17884 sys_catalog.cc:564] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:53.975888 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b547e792b86941398e04e3f6fb122765. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } }
I20260621 02:50:53.976557 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:53.976471 17890 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } }
I20260621 02:50:53.977208 17890 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:53.978854 17898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:50:53.983093 17898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:50:53.990669 17898 catalog_manager.cc:1347] Generated new cluster ID: 29e1e9b8c1bf476fbc48745f001454d0
I20260621 02:50:53.990765 17898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:50:54.010830 17898 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:50:54.012048 17898 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:50:54.020466 17898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Generated new TSK 0
I20260621 02:50:54.021272 17898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab.
Entry for principal oryx/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab.
I20260621 02:50:54.070413 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36053
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:54.383877 17910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:54.384135 17910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:54.384230 17910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:54.391112 17910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:54.391211 17910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:54.391319 17910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:54.400467 17910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:54.401764 17910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:54.403048 17910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:54.415349 17915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:54.415428 17916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:54.415388 17918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:54.416335 17910 server_base.cc:1034] running on GCE node
I20260621 02:50:54.416900 17910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:54.417783 17910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:54.418995 17910 hybrid_clock.cc:648] HybridClock initialized: now 1782010254418958 us; error 46 us; skew 500 ppm
I20260621 02:50:54.419288 17910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:54 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010254, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:54.422241 17910 init.cc:377] Logged in from keytab as oryx/127.11.109.1@KRBTEST.COM (short username oryx)
I20260621 02:50:54.423544 17910 webserver.cc:460] Webserver started at http://127.11.109.1:33969/ using document root <none> and password file <none>
I20260621 02:50:54.424170 17910 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:54.424304 17910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:54.424589 17910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:54.427528 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "58ba662c9ec34515bc826593508a9c48"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "bc7f0700dabf5298101e8ca938fa0ef7"
server_key_iv: "8cda73eb761b09586518c170aba00992"
server_key_version: "encryptionkey@0"
I20260621 02:50:54.428285 17910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "58ba662c9ec34515bc826593508a9c48"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "bc7f0700dabf5298101e8ca938fa0ef7"
server_key_iv: "8cda73eb761b09586518c170aba00992"
server_key_version: "encryptionkey@0"
I20260621 02:50:54.433146 17910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260621 02:50:54.436357 17925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:54.437644 17910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:54.437809 17910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "58ba662c9ec34515bc826593508a9c48"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "bc7f0700dabf5298101e8ca938fa0ef7"
server_key_iv: "8cda73eb761b09586518c170aba00992"
server_key_version: "encryptionkey@0"
I20260621 02:50:54.437999 17910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:54.491117 17910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:54.495020 17910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:54.495287 17910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:54.496201 17910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:54.497548 17910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:54.497642 17910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:54.497772 17910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:54.497838 17910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:54.529402 17910 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:35299
I20260621 02:50:54.529476 18037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:35299 every 8 connection(s)
I20260621 02:50:54.530620 17910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:54.537684 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 17910
I20260621 02:50:54.537882 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:50:54.538535 11700 external_mini_cluster.cc:1428] Setting key 96552d2af09578b23a34a68312d024dd
Jun 21 02:50:54 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010254, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:54.550122 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:54.532996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:60681 (local address 127.11.109.62:36053)
0621 02:50:54.533441 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:54.533446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:54.534367 (+ 921us) server_negotiation.cc:407] Connection header received
0621 02:50:54.536050 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:54.536056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:54.536148 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:54.536269 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:54.537759 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:54.538532 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:54.540229 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:54.540614 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:54.540754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:54.544393 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:54.544398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:54.544401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:54.544430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:54.546938 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:54.547880 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:54.547884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:54.547886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:54.548008 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:54.548442 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:54.548445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:54.548447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:54.548694 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:54.548797 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:54.549713 (+ 916us) server_negotiation.cc:299] Negotiation successful
0621 02:50:54.549873 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":262,"threads_started":1}
I20260621 02:50:54.551569 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:54.533547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:60681)
0621 02:50:54.534223 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:54.534258 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:54.535797 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:54.536486 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:54.536494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:54.536976 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:54.537601 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:54.537618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:54.538683 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:54.538688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:54.540087 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:54.540105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:54.540835 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:54.540844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:54.541093 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:54.541831 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:54.541852 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:54.544200 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:54.547070 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:54.547082 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:54.547101 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:54.547752 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:54.548113 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:54.548121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:54.548125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:54.548340 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:54.548814 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:54.548825 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:54.549300 (+ 475us) client_negotiation.cc:769] Sending connection context
0621 02:50:54.549614 (+ 314us) client_negotiation.cc:240] Negotiation successful
0621 02:50:54.550162 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260621 02:50:54.553071 18038 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36053
I20260621 02:50:54.553378 18038 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:54.554069 18038 heartbeater.cc:511] Master 127.11.109.62:36053 requested a full tablet report, sending...
I20260621 02:50:54.556370 17849 ts_manager.cc:194] Registered new tserver with Master: 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:50:54.558549 17849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:60681
WARNING: no policy specified for oryx/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab.
Entry for principal oryx/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab.
I20260621 02:50:54.584456 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36053
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:54.894572 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:54.894856 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:54.894948 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:54.902172 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:54.902272 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:54.902395 18044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:54.911569 18044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:54.912676 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:54.913877 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:54.926244 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:54.926342 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:54.926324 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:54.926894 18044 server_base.cc:1034] running on GCE node
I20260621 02:50:54.927505 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:54.928339 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:54.929548 18044 hybrid_clock.cc:648] HybridClock initialized: now 1782010254929513 us; error 45 us; skew 500 ppm
I20260621 02:50:54.929823 18044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:54 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010254, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:54.932682 18044 init.cc:377] Logged in from keytab as oryx/127.11.109.2@KRBTEST.COM (short username oryx)
I20260621 02:50:54.934152 18044 webserver.cc:460] Webserver started at http://127.11.109.2:44403/ using document root <none> and password file <none>
I20260621 02:50:54.934798 18044 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:54.934928 18044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:54.935199 18044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:54.938040 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "61b1604bfed0596e7375168eb33bc388"
server_key_iv: "4bd6157e4a4599eeebe8921693de6ff7"
server_key_version: "encryptionkey@0"
I20260621 02:50:54.938807 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "61b1604bfed0596e7375168eb33bc388"
server_key_iv: "4bd6157e4a4599eeebe8921693de6ff7"
server_key_version: "encryptionkey@0"
I20260621 02:50:54.943910 18044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:50:54.947188 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:54.948359 18044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:50:54.948516 18044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "61b1604bfed0596e7375168eb33bc388"
server_key_iv: "4bd6157e4a4599eeebe8921693de6ff7"
server_key_version: "encryptionkey@0"
I20260621 02:50:54.948696 18044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:54.971899 18044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:54.975874 18044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:54.976163 18044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:54.977108 18044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:54.978430 18044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:54.978531 18044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:54.978663 18044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:54.978729 18044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:55.010597 18044 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:36383
I20260621 02:50:55.010625 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:36383 every 8 connection(s)
I20260621 02:50:55.011922 18044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:55.021622 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18044
I20260621 02:50:55.021822 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:50:55.022302 11700 external_mini_cluster.cc:1428] Setting key 4b9b4a61d4fa7344595f3ca49911e9a2
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010254, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:55.031251 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.014456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:53483 (local address 127.11.109.62:36053)
0621 02:50:55.014629 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:55.014634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:55.015745 (+ 1111us) server_negotiation.cc:407] Connection header received
0621 02:50:55.017595 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:55.017600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:55.017704 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:55.017812 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:55.019224 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.019951 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.021719 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.022171 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.022270 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.025824 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:55.025829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:55.025832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:55.025861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:55.028360 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.029062 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.029068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.029070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.029240 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.029640 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.029645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.029648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.029904 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:55.030053 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:55.030705 (+ 652us) server_negotiation.cc:299] Negotiation successful
0621 02:50:55.030917 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:50:55.032181 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.014924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:53483)
0621 02:50:55.015598 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:55.015635 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:55.017335 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:55.018038 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:55.018046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:55.018523 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:55.019059 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.019076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.020112 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.020116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:55.021536 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.021552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.022388 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.022397 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:55.022675 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.023388 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:55.023410 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:55.025615 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:55.028497 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:55.028505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:55.028521 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:55.028938 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:55.029340 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:55.029346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:55.029349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:55.029526 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:55.030030 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:55.030038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:55.030351 (+ 313us) client_negotiation.cc:769] Sending connection context
0621 02:50:55.030608 (+ 257us) client_negotiation.cc:240] Negotiation successful
0621 02:50:55.031118 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":258,"threads_started":1}
I20260621 02:50:55.033599 18172 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36053
I20260621 02:50:55.033882 18172 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:55.034549 18172 heartbeater.cc:511] Master 127.11.109.62:36053 requested a full tablet report, sending...
I20260621 02:50:55.035989 17849 ts_manager.cc:194] Registered new tserver with Master: e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383)
I20260621 02:50:55.037304 17849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:53483
WARNING: no policy specified for oryx/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab.
Entry for principal oryx/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab.
I20260621 02:50:55.065989 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36053
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:55.380445 18177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:55.380707 18177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:55.380800 18177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:55.387933 18177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:55.388037 18177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:55.388154 18177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:55.397315 18177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:55.398408 18177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:55.399621 18177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:55.412031 18185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:55.412058 18182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:55.411999 18183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:55.413115 18177 server_base.cc:1034] running on GCE node
I20260621 02:50:55.413786 18177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:55.414615 18177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:55.415827 18177 hybrid_clock.cc:648] HybridClock initialized: now 1782010255415794 us; error 45 us; skew 500 ppm
I20260621 02:50:55.416108 18177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010255, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:55.419334 18177 init.cc:377] Logged in from keytab as oryx/127.11.109.3@KRBTEST.COM (short username oryx)
I20260621 02:50:55.420815 18177 webserver.cc:460] Webserver started at http://127.11.109.3:45577/ using document root <none> and password file <none>
I20260621 02:50:55.421562 18177 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:55.421687 18177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:55.422000 18177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:55.424728 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "2a15586458ed42b2ae80e2dbc4b6d669"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "24e0658ab253f9b55e6e9338bdaee528"
server_key_iv: "172418a63810564e4f9dbcdfe2d8d0de"
server_key_version: "encryptionkey@0"
I20260621 02:50:55.425550 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "2a15586458ed42b2ae80e2dbc4b6d669"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "24e0658ab253f9b55e6e9338bdaee528"
server_key_iv: "172418a63810564e4f9dbcdfe2d8d0de"
server_key_version: "encryptionkey@0"
I20260621 02:50:55.430294 18177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:50:55.433559 18192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:55.434774 18177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260621 02:50:55.434934 18177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "2a15586458ed42b2ae80e2dbc4b6d669"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "24e0658ab253f9b55e6e9338bdaee528"
server_key_iv: "172418a63810564e4f9dbcdfe2d8d0de"
server_key_version: "encryptionkey@0"
I20260621 02:50:55.435133 18177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:55.453835 18177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:55.457724 18177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:55.458021 18177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:55.458954 18177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:55.460229 18177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:55.460327 18177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:55.460469 18177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:55.460536 18177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:55.492213 18177 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35659
I20260621 02:50:55.492240 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35659 every 8 connection(s)
I20260621 02:50:55.493520 18177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:55.503489 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18177
I20260621 02:50:55.503659 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:50:55.504153 11700 external_mini_cluster.cc:1428] Setting key 0eca4fa09879d39f7444b9129784cf02
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010255, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:55.512383 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.495863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:51107 (local address 127.11.109.62:36053)
0621 02:50:55.496031 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:55.496035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:55.497212 (+ 1177us) server_negotiation.cc:407] Connection header received
0621 02:50:55.499058 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:55.499064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:55.499175 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:55.499283 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:55.500676 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.501525 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.503034 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.503695 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.503825 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.507455 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:55.507459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:55.507463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:55.507489 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:55.509830 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.510499 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.510502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.510505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.510635 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.511088 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.511091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.511093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.511330 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:55.511448 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:55.512002 (+ 554us) server_negotiation.cc:299] Negotiation successful
0621 02:50:55.512141 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:50:55.513343 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.496367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:51107)
0621 02:50:55.497044 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:55.497079 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:55.498807 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:55.499502 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:55.499510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:55.499971 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:55.500518 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.500533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.501660 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.501665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:55.502907 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.502917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.503907 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.503913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:55.504150 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.504905 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:55.504930 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:55.507295 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:55.509946 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:55.509952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:55.509965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:55.510367 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:55.510817 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:55.510821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:55.510823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:55.510995 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:55.511447 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:55.511453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:55.511723 (+ 270us) client_negotiation.cc:769] Sending connection context
0621 02:50:55.511932 (+ 209us) client_negotiation.cc:240] Negotiation successful
0621 02:50:55.512252 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260621 02:50:55.514488 18305 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36053
I20260621 02:50:55.514776 18305 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:55.515406 18305 heartbeater.cc:511] Master 127.11.109.62:36053 requested a full tablet report, sending...
I20260621 02:50:55.516650 17849 ts_manager.cc:194] Registered new tserver with Master: 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659)
I20260621 02:50:55.517607 17849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:51107
I20260621 02:50:55.519145 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.61.keytab.
Entry for principal oryx/127.11.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.61.keytab.
I20260621 02:50:55.561602 18038 heartbeater.cc:503] Master 127.11.109.62:36053 was elected leader, sending a full tablet report...
I20260621 02:50:55.562796 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:42527
--webserver_interface=127.11.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.61.keytab
--principal=oryx/127.11.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:36053,127.11.109.61:42527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:55.869421 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:55.869681 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:55.869863 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:55.877064 18310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:55.877207 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:55.877311 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:55.877413 18310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:55.877490 18310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:55.886888 18310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:36053,127.11.109.61:42527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.61.keytab
--principal=oryx/127.11.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:42527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
W20260621 02:50:55.887229 18310 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.109.62:36053,127.11.109.61:42527'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 02:50:55.888278 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:55.889565 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:55.900452 18318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:55.900554 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:55.900573 18315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:55.901539 18310 server_base.cc:1034] running on GCE node
I20260621 02:50:55.902323 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:55.903596 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:55.904810 18310 hybrid_clock.cc:648] HybridClock initialized: now 1782010255904737 us; error 51 us; skew 500 ppm
I20260621 02:50:55.905115 18310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010255, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:55.908005 18310 init.cc:377] Logged in from keytab as oryx/127.11.109.61@KRBTEST.COM (short username oryx)
I20260621 02:50:55.909404 18310 webserver.cc:460] Webserver started at http://127.11.109.61:40825/ using document root <none> and password file <none>
I20260621 02:50:55.910101 18310 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:55.910228 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:55.910564 18310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:55.913412 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/instance:
uuid: "d4aa724987c54feab6038e6ce99b67f5"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "d05a5eb452e8c1e009818a559175cc1b"
server_key_iv: "af5c76efd612e7420d309ab1bbc738f8"
server_key_version: "encryptionkey@0"
I20260621 02:50:55.914160 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal/instance:
uuid: "d4aa724987c54feab6038e6ce99b67f5"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "d05a5eb452e8c1e009818a559175cc1b"
server_key_iv: "af5c76efd612e7420d309ab1bbc738f8"
server_key_version: "encryptionkey@0"
I20260621 02:50:55.919092 18310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:50:55.922451 18325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:55.923653 18310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:50:55.923825 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
uuid: "d4aa724987c54feab6038e6ce99b67f5"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "d05a5eb452e8c1e009818a559175cc1b"
server_key_iv: "af5c76efd612e7420d309ab1bbc738f8"
server_key_version: "encryptionkey@0"
I20260621 02:50:55.924023 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:55.942978 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:55.946432 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:55.946668 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:55.963881 18310 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.61:42527
I20260621 02:50:55.963900 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.61:42527 every 8 connection(s)
I20260621 02:50:55.965157 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
I20260621 02:50:55.968755 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:55.969411 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18310
I20260621 02:50:55.978037 18377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010255, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.61@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:55.994381 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.971427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.11.109.61:42527)
0621 02:50:55.972151 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:55.972166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:55.972214 (+ 48us) server_negotiation.cc:407] Connection header received
0621 02:50:55.975037 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:55.975064 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:55.975533 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:55.976004 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:55.977606 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.979483 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.980926 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.982028 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.982188 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.986572 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:55.986579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:55.986588 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:55.986619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:55.990168 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.990880 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.990888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.990896 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.991072 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.991482 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.991487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.991490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.992174 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:55.992448 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:55.992777 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 02:50:55.993190 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":300,"threads_started":1}
I20260621 02:50:55.996448 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.980693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.11.109.62:36053)
0621 02:50:55.980837 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:55.980840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:55.981674 (+ 834us) server_negotiation.cc:407] Connection header received
0621 02:50:55.981840 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:55.981844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:55.981904 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:55.981983 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:55.983306 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.984547 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.985933 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.986596 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.986688 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.990494 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:55.990498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:55.990501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:55.990529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:55.993690 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.994387 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.994394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.994401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.994552 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:55.994944 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:55.994951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:55.994955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:55.995362 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:55.995511 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:55.995924 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 02:50:55.996135 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:50:55.996580 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.980731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40220)
0621 02:50:55.981509 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:55.981566 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:55.981731 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:55.982056 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:55.982065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:55.982924 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:55.983158 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.983165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.984680 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.984683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:55.985797 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.985808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:55.986768 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:55.986774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:55.986971 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:55.987676 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:55.987698 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:55.990329 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:55.993808 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:55.993816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:55.993820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:55.994254 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:55.994669 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:55.994675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:55.994679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:55.994826 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:55.995490 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:55.995495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:55.995638 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 02:50:55.995822 (+ 184us) client_negotiation.cc:240] Negotiation successful
0621 02:50:55.996125 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":383,"threads_started":1}
I20260621 02:50:55.998128 18377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:50:55 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010255, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.61@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
I20260621 02:50:56.009231 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.998716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44610)
0621 02:50:55.998890 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:55.998905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:55.998975 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:55.999204 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:55.999207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:55.999500 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:55.999634 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:55.999641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.000496 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.000500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.001530 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.001541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.002131 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.002135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.002314 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.002778 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.002794 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.004590 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.007243 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.007247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.007254 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.007635 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.007911 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.007914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.007916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.008031 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.008462 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.008465 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.008597 (+ 132us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.008700 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.008912 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 02:50:56.009438 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:55.998839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.11.109.61:42527)
0621 02:50:55.998975 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:55.998979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:55.998993 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:50:55.999056 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:55.999060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:55.999106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:55.999183 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:55.999750 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.000389 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.001635 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.002009 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.002102 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.004743 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.004748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.004751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.004776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.007117 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.007730 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.007733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.007735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.007829 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.008110 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.008113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.008115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.008365 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.008462 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.008788 (+ 326us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.009032 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":49}
I20260621 02:50:56.015106 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Bootstrap starting.
I20260621 02:50:56.018412 18377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:56.019333 18377 log.cc:824] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:56.021754 18377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: No bootstrap required, opened a new log
I20260621 02:50:56.028360 18377 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } }
I20260621 02:50:56.028625 18377 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:56.028708 18377 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4aa724987c54feab6038e6ce99b67f5, State: Initialized, Role: FOLLOWER
I20260621 02:50:56.029346 18377 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } }
I20260621 02:50:56.030071 18383 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } } }
I20260621 02:50:56.030387 18383 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:50:56.030997 18377 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:50:56.039563 18394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:50:56.039686 18394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:50:56.039709 18172 heartbeater.cc:503] Master 127.11.109.62:36053 was elected leader, sending a full tablet report...
I20260621 02:50:56.041889 18310 master_runner.cc:428] Detected that this master d4aa724987c54feab6038e6ce99b67f5 is joining an existing cluster
I20260621 02:50:56.041996 18310 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.109.61:42527
I20260621 02:50:56.063812 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.045805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40236)
0621 02:50:56.046467 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.046487 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.046649 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.047314 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.047321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.048282 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.048754 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.048770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.050573 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.050581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.052820 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.052840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.054416 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.054425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.054658 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.055376 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.055392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.056267 (+ 875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.060331 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.060343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.060349 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.060915 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.061405 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.061412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.061419 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.061617 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.062332 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.062340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.062590 (+ 250us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.062783 (+ 193us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.063222 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":365,"threads_started":1}
I20260621 02:50:56.064510 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.045770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.11.109.62:36053)
0621 02:50:56.045948 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.045957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.046556 (+ 599us) server_negotiation.cc:407] Connection header received
0621 02:50:56.046810 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.046818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.046944 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.047048 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.048957 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.050371 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.053033 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.053925 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.054061 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.056476 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.056484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.056489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.056528 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.060185 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.061072 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.061080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.061086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.061267 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.061748 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.061755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.061761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.062132 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.062264 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.063917 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.064158 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for oryx/127.11.109.60@KRBTEST.COM; defaulting to no policy
I20260621 02:50:56.072666 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.046716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.11.109.61:42527)
0621 02:50:56.046902 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.046925 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.057564 (+ 10639us) server_negotiation.cc:407] Connection header received
0621 02:50:56.057765 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.057770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.057835 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.057895 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.059093 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.060139 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.062178 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.062888 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.062995 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.065992 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.065998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.066002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.066040 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.069554 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.070346 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.070351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.070354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.070491 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.070901 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.070909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.070914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.071239 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.071500 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.071967 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.072286 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":50}
I20260621 02:50:56.072700 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.046629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44620)
0621 02:50:56.057478 (+ 10849us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.057498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.057641 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.058057 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.058061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.058664 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.058954 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.058964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.060394 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.060400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.062025 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.062039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.063299 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.063306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.063495 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.064785 (+ 1290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.064818 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.065814 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.069688 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.069693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.069696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.070227 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.070619 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.070623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.070625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.070759 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.071563 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.071571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.071728 (+ 157us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.071878 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.072156 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10712,"thread_start_us":8509,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.60@KRBTEST.COM" created.
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.62@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
I20260621 02:50:56.091827 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.071397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.11.109.61:42527)
0621 02:50:56.073038 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.073044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.073060 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:50:56.073136 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.073139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.073342 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.073440 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.076057 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.077253 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.079398 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.080289 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.080431 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.085080 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.085087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.085096 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.085131 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.088615 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.089540 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.089545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.089549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.089695 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.090131 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.090136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.090140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.090444 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.090584 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.091210 (+ 626us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.091426 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":15384,"threads_started":1}
I20260621 02:50:56.091923 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.070995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44632)
0621 02:50:56.071531 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.071586 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.071726 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.074550 (+ 2824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.074562 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.075557 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.075874 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.075889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.077378 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.077386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.079212 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.079227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.080465 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.080471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.080681 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.081554 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.081586 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.084884 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.088746 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.088760 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.088770 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.089414 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.089823 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.089831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.089837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.090006 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.090599 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.090610 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.090859 (+ 249us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.091133 (+ 274us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.091491 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":666,"threads_started":1}
I20260621 02:50:56.094132 17849 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.109.61:42527: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5"
member_type: NON_VOTER
last_known_addr {
host: "127.11.109.61"
port: 42527
}
attrs {
promote: true
}
}
dest_uuid: "b547e792b86941398e04e3f6fb122765"
cas_config_opid_index: -1
I20260621 02:50:56.095273 17849 consensus_queue.cc:237] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } }
I20260621 02:50:56.098320 18407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d4aa724987c54feab6038e6ce99b67f5 (127.11.109.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } }
I20260621 02:50:56.099262 18352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.60.keytab.
Entry for principal oryx/127.11.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.60.keytab.
I20260621 02:50:56.102644 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:43443
--webserver_interface=127.11.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.60.keytab
--principal=oryx/127.11.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:50:56.102723 18407 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.109.61:42527
I20260621 02:50:56.103042 18352 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 02:50:56.103113 18407 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } }
I20260621 02:50:56.103338 18407 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:56.105777 18310 master.cc:537] Master@127.11.109.61:42527 shutting down...
I20260621 02:50:56.106829 18408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:50:56.109918 17836 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260621 02:50:56.116091 17836 consensus_peers.cc:487] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 -> Peer d4aa724987c54feab6038e6ce99b67f5 (127.11.109.61:42527): Couldn't send request to peer d4aa724987c54feab6038e6ce99b67f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 02:50:56.119077 18310 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:56.119486 18310 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:56.119673 18310 tablet_replica.cc:330] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: stopping tablet replica
I20260621 02:50:56.126884 18310 master.cc:552] Master@127.11.109.61:42527 shutdown complete.
I20260621 02:50:56.126968 18310 master_runner.cc:305] Clearing existing system tablet
I20260621 02:50:56.127399 18310 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:56.127498 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:56.130426 18310 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260621 02:50:56.135972 18413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:56.136317 18310 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:56.136746 18310 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:50:56.136879 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
uuid: "d4aa724987c54feab6038e6ce99b67f5"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "d05a5eb452e8c1e009818a559175cc1b"
server_key_iv: "af5c76efd612e7420d309ab1bbc738f8"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.142098 18310 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:56.149056 18310 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 02:50:56.149390 18310 log.cc:1196] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 02:50:56.149931 18310 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Deleting consensus metadata
I20260621 02:50:56.150254 18310 master_runner.cc:315] Copying system tablet from 127.11.109.62:36053
I20260621 02:50:56.152652 18310 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: tablet copy: Beginning tablet copy session from remote peer at address 127.11.109.62:36053
I20260621 02:50:56.180514 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.153509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.11.109.62:36053)
0621 02:50:56.153695 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.153702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.161779 (+ 8077us) server_negotiation.cc:407] Connection header received
0621 02:50:56.161972 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.161980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.162098 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.162246 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.163368 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.164670 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.166595 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.167196 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.167328 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.174225 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.174232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.174236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.174276 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.177719 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.178474 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.178487 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.178490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.178628 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.179022 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.179027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.179030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.179364 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.179527 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.179960 (+ 433us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.180190 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:50:56.181059 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.153500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40248)
0621 02:50:56.161620 (+ 8120us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.161643 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.161821 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.162291 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.162296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.162893 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.163190 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.163203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.164828 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.164846 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.166442 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.166457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.167358 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.167365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.167553 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.172804 (+ 5251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.172827 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.174006 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.177837 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.177865 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.177869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.178335 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.178737 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.178742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.178746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.178899 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.179482 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.179487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.179660 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.179821 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.180109 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7943,"thread_start_us":7852,"threads_started":1}
I20260621 02:50:56.181941 17869 tablet_copy_service.cc:140] P b547e792b86941398e04e3f6fb122765: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d4aa724987c54feab6038e6ce99b67f5 ({username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:40248)
I20260621 02:50:56.182219 17869 tablet_copy_service.cc:161] P b547e792b86941398e04e3f6fb122765: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d4aa724987c54feab6038e6ce99b67f5 at {username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:40248: session id = d4aa724987c54feab6038e6ce99b67f5-00000000000000000000000000000000
I20260621 02:50:56.186710 17869 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 02:50:56.188818 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 02:50:56.195422 18310 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: tablet copy: Starting download of 0 data blocks...
I20260621 02:50:56.195717 18310 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: tablet copy: Starting download of 1 WAL segments...
I20260621 02:50:56.198227 18310 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 02:50:56.202210 17869 tablet_copy_service.cc:342] P b547e792b86941398e04e3f6fb122765: Request end of tablet copy session d4aa724987c54feab6038e6ce99b67f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:40248
I20260621 02:50:56.202402 17869 tablet_copy_service.cc:433] P b547e792b86941398e04e3f6fb122765: ending tablet copy session d4aa724987c54feab6038e6ce99b67f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d4aa724987c54feab6038e6ce99b67f5
I20260621 02:50:56.212422 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:56.215082 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:56.221405 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:56.225059 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:56.225682 18310 server_base.cc:1034] running on GCE node
I20260621 02:50:56.225975 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:56.226547 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:56.227751 18310 hybrid_clock.cc:648] HybridClock initialized: now 1782010256227679 us; error 52 us; skew 500 ppm
I20260621 02:50:56.227947 18310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010256, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:56.229743 18310 init.cc:377] Logged in from keytab as oryx/127.11.109.61@KRBTEST.COM (short username oryx)
I20260621 02:50:56.230811 18310 webserver.cc:460] Webserver started at http://127.11.109.61:33487/ using document root <none> and password file <none>
I20260621 02:50:56.231242 18310 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:56.231345 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:56.233975 18310 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:50:56.235499 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:56.235883 18310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 02:50:56.236012 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
uuid: "d4aa724987c54feab6038e6ce99b67f5"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "d05a5eb452e8c1e009818a559175cc1b"
server_key_iv: "af5c76efd612e7420d309ab1bbc738f8"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.236164 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:56.251422 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:56.254999 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:56.282851 18310 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.61:42527
I20260621 02:50:56.282873 18481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.61:42527 every 8 connection(s)
I20260621 02:50:56.283747 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
I20260621 02:50:56.290596 18482 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:50:56.292969 18482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Bootstrap starting.
I20260621 02:50:56.305377 18482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:50:56.305795 18482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Bootstrap complete.
I20260621 02:50:56.306627 18482 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } }
I20260621 02:50:56.306880 18482 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d4aa724987c54feab6038e6ce99b67f5, State: Initialized, Role: LEARNER
I20260621 02:50:56.307099 18482 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } }
I20260621 02:50:56.307942 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } }
I20260621 02:50:56.308318 18482 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:56.308347 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:50:56.316689 18496 catalog_manager.cc:1259] Loaded cluster ID: 29e1e9b8c1bf476fbc48745f001454d0
I20260621 02:50:56.316789 18496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: loading cluster ID for follower catalog manager: success
I20260621 02:50:56.319106 18496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: acquiring CA information for follower catalog manager: success
I20260621 02:50:56.320801 18496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:50:56.491979 18411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:56.492302 18411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:56.492473 18411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:56.500824 18411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:50:56.500944 18411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:56.501056 18411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:56.501153 18411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:50:56.501264 18411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:50:56.511633 18411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.60.keytab
--principal=oryx/127.11.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:56.512986 18411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:56.514289 18411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:50:56.520148 18305 heartbeater.cc:503] Master 127.11.109.62:36053 was elected leader, sending a full tablet report...
W20260621 02:50:56.526525 18502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:56.526764 18504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:56.526743 18501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:56.527659 18411 server_base.cc:1034] running on GCE node
I20260621 02:50:56.528381 18411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:56.529875 18411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:56.531082 18411 hybrid_clock.cc:648] HybridClock initialized: now 1782010256531005 us; error 71 us; skew 500 ppm
I20260621 02:50:56.531396 18411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010256, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:56.534824 18411 init.cc:377] Logged in from keytab as oryx/127.11.109.60@KRBTEST.COM (short username oryx)
I20260621 02:50:56.536512 18411 webserver.cc:460] Webserver started at http://127.11.109.60:39353/ using document root <none> and password file <none>
I20260621 02:50:56.537355 18411 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:56.537489 18411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:56.537817 18411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:50:56.540799 18411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/instance:
uuid: "bae160b446ca4b27b3b2693d5141436d"
format_stamp: "Formatted at 2026-06-21 02:50:56 on dist-test-slave-bc52"
server_key: "a6cc8f5055af17b842706aba4d88118e"
server_key_iv: "55d84011a9eb55303707ddebf5e78d0a"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.541896 18411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal/instance:
uuid: "bae160b446ca4b27b3b2693d5141436d"
format_stamp: "Formatted at 2026-06-21 02:50:56 on dist-test-slave-bc52"
server_key: "a6cc8f5055af17b842706aba4d88118e"
server_key_iv: "55d84011a9eb55303707ddebf5e78d0a"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.546912 18411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:50:56.550194 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:56.551535 18411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:56.551693 18411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
uuid: "bae160b446ca4b27b3b2693d5141436d"
format_stamp: "Formatted at 2026-06-21 02:50:56 on dist-test-slave-bc52"
server_key: "a6cc8f5055af17b842706aba4d88118e"
server_key_iv: "55d84011a9eb55303707ddebf5e78d0a"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.551869 18411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:56.572588 18411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:56.574677 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.568222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44636)
0621 02:50:56.568411 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.568430 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.568541 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.569754 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.569759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.569789 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:56.569962 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.569971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.571274 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.571279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.572502 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.572514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.573855 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.573860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.574057 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.574144 (+ 87us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.574255 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.574394 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:50:56.574729 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.568449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.11.109.61:42527)
0621 02:50:56.569404 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.569408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.569424 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:50:56.569480 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.569484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.569553 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.569700 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:56.570135 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.571124 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.572649 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.573640 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.573816 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.574044 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.574277 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.574401 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":728,"threads_started":1}
I20260621 02:50:56.575668 18457 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260621 02:50:56.576253 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b547e792b86941398e04e3f6fb122765. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } }
I20260621 02:50:56.576504 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:50:56.577618 18411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:56.577893 18411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:56.597149 18411 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.60:43443
I20260621 02:50:56.597230 18563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.60:43443 every 8 connection(s)
I20260621 02:50:56.598448 18411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
I20260621 02:50:56.600495 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18411
I20260621 02:50:56.602470 18564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:56.611969 18564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.60@KRBTEST.COM
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010256, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.60@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:56.625918 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.602430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.11.109.60:43443)
0621 02:50:56.603234 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.603245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.605654 (+ 2409us) server_negotiation.cc:407] Connection header received
0621 02:50:56.607494 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.607515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.607872 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.608247 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.609880 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.611250 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.612840 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.613848 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.614028 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.618942 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.618950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.618959 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.618992 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.622011 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.622669 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.622675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.622680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.622867 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.623305 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.623308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.623310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.623856 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.624092 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.624523 (+ 431us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.624865 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":378,"threads_started":1}
I20260621 02:50:56.627836 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.614692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.11.109.62:36053)
0621 02:50:56.614888 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.614896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.615374 (+ 478us) server_negotiation.cc:407] Connection header received
0621 02:50:56.615560 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.615566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.615629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.615723 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.616728 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.617474 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.618772 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.619164 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.619284 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.622297 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.622302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.622305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.622334 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.625269 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.625928 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.625936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.625940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.626096 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.626532 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.626537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.626540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.626795 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.626949 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.627450 (+ 501us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.627604 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:50:56.628047 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.614548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40252)
0621 02:50:56.615254 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.615289 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.615445 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.615776 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.615784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.616348 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.616591 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.616600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.617594 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.617601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.618631 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.618642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.619341 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.619352 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.619513 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.619991 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.620008 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.622127 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.625382 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.625394 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.625401 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.625801 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.626277 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.626282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.626285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.626415 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.626918 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.626925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.627132 (+ 207us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.627367 (+ 235us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.627667 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":390,"threads_started":1}
I20260621 02:50:56.629295 18564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010256, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.60@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
I20260621 02:50:56.641644 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.629863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44652)
0621 02:50:56.630017 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.630037 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.630106 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.630400 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.630403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.630686 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.630801 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.630808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.631700 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.631704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.632632 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.632643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.633317 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.633322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.633457 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.633945 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.633963 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.635919 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.639384 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.639387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.639389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.639753 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.640211 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.640215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.640217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.640352 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.640798 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.640801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.640915 (+ 114us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.641146 (+ 231us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.641429 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260621 02:50:56.641708 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.630020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.11.109.61:42527)
0621 02:50:56.630175 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.630179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.630193 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:50:56.630239 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.630242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.630297 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.630373 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.630912 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.631588 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.632754 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.633131 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.633294 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.636082 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.636086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.636089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.636117 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.639265 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.639858 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.639862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.639864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.639978 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.640450 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.640453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.640454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.640696 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.640800 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.641269 (+ 469us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.641478 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:50:56.642746 18564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010256, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.60@KRBTEST.COM for oryx/127.11.109.60@KRBTEST.COM
I20260621 02:50:56.654565 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.643368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.0.0.1:38496)
0621 02:50:56.643578 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.643605 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.643677 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.643907 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.643910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.644242 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.644372 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.644379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.645282 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.645285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.646095 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.646105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.647032 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.647041 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.647301 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.647911 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.647927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.649794 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.652442 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.652445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.652448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.652933 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.653286 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.653289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.653291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.653426 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.653879 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.653882 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.653998 (+ 116us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.654115 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.654286 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:50:56.654636 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.643566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.11.109.60:43443)
0621 02:50:56.643706 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.643710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.643723 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:50:56.643758 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.643761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.643808 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.643879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.644478 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.645118 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.646215 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.646833 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.646971 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.649939 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.649943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.649947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.649973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.652312 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.653038 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.653042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.653044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.653143 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.653511 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.653515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.653517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.653779 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.653881 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.654188 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.654396 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":45}
I20260621 02:50:56.659348 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Bootstrap starting.
I20260621 02:50:56.662801 18564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:56.663805 18564 log.cc:824] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:56.666287 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: No bootstrap required, opened a new log
I20260621 02:50:56.673089 18564 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } }
I20260621 02:50:56.673475 18564 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:56.673585 18564 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae160b446ca4b27b3b2693d5141436d, State: Initialized, Role: FOLLOWER
I20260621 02:50:56.674170 18564 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } }
I20260621 02:50:56.675158 18570 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } } }
I20260621 02:50:56.675490 18570 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:50:56.676084 18564 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:50:56.686237 18581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:50:56.686753 18581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:50:56.690269 18411 master_runner.cc:428] Detected that this master bae160b446ca4b27b3b2693d5141436d is joining an existing cluster
I20260621 02:50:56.690366 18411 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.109.60:43443
I20260621 02:50:56.712332 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.694978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.11.109.61:42527)
0621 02:50:56.695140 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.695145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.695730 (+ 585us) server_negotiation.cc:407] Connection header received
0621 02:50:56.695927 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.695934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.696005 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.696076 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.697556 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.698705 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.700842 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.701610 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.701698 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.705344 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.705351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.705355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.705397 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.709092 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.709936 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.709941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.709945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.710091 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.710572 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.710577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.710580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.710970 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.711108 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.711758 (+ 650us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.711993 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 02:50:56.713380 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.693960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.11.109.62:36053)
0621 02:50:56.694126 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.694130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.694728 (+ 598us) server_negotiation.cc:407] Connection header received
0621 02:50:56.694948 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.694953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.695034 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.695104 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.697611 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.698939 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.700883 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.701593 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.701712 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.705293 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.705302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.705310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.705349 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.708906 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.709742 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.709747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.709751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.709944 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.710487 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.710493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.710497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.710897 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.711094 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.712658 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.712913 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:50:56.714135 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.694045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40268)
0621 02:50:56.694627 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.694644 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.694789 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.695350 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.695358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.696919 (+ 1561us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.697420 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.697430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.699118 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.699124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.700692 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.700701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.701931 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.701938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.702147 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.703076 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.703107 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.705039 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.709064 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.709074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.709080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.709603 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.710098 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.710105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.710114 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.710333 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.711160 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.711165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.711437 (+ 272us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.711664 (+ 227us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.711984 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":138,"spinlock_wait_cycles":36736,"thread_start_us":287,"threads_started":1}
I20260621 02:50:56.714670 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.694904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44658)
0621 02:50:56.695641 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.695664 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.695815 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.696318 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.696323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.696968 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.697373 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.697387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.698897 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.698902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.700676 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.700690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.702043 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.702050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.702275 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.703253 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.703278 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.705081 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.709278 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.709283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.709288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.709805 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.710241 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.710245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.710252 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.710441 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.711203 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.711210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.711449 (+ 239us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.712861 (+ 1412us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.713186 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":45,"thread_start_us":427,"threads_started":1}
I20260621 02:50:56.718566 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.696131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.11.109.60:43443)
0621 02:50:56.696303 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.696310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.703177 (+ 6867us) server_negotiation.cc:407] Connection header received
0621 02:50:56.703389 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.703396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.703463 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.703544 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.705328 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.706434 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.708549 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.709284 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.709384 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.711802 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.711808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.711812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.711846 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.715449 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.716225 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.716230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.716233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.716392 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.716803 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.716810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.716816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.717207 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.717378 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.717860 (+ 482us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.718209 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":54}
I20260621 02:50:56.718588 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.696406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.0.0.1:38502)
0621 02:50:56.703087 (+ 6681us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.703108 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.703261 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.703750 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.703756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.704366 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.705196 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.705211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.706672 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.706677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.708392 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.708410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.709723 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.709729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.709912 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.710482 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.710522 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.711656 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.715603 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.715608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.715612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.716103 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.716527 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.716546 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.716550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.716693 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.717433 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.717439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.717569 (+ 130us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.717760 (+ 191us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.718087 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6434,"thread_start_us":6247,"threads_started":1}
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.62@KRBTEST.COM for oryx/127.11.109.60@KRBTEST.COM
I20260621 02:50:56.732074 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17910
I20260621 02:50:56.732997 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.720137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.0.0.1:38514)
0621 02:50:56.720496 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.720512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.720614 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.721035 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.721040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.721425 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.721570 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.721578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.722487 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.722494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.723319 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.723329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.724633 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.724639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.724806 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.725356 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.725373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.727496 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.730382 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.730389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.730395 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.730832 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.731270 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.731276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.731280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.731431 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.732042 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.732047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.732209 (+ 162us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.732367 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.732627 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260621 02:50:56.733011 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.720541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.11.109.60:43443)
0621 02:50:56.720715 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.720721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.720735 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:50:56.720809 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.720812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.720884 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.720997 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.721716 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.722366 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.723471 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.724486 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.724619 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.727664 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.727669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.727672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.727698 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.730244 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.731006 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.731010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.731012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.731144 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.731550 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.731556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.731559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.731918 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.732074 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.732477 (+ 403us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.732689 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:50:56.735138 17849 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.109.60:43443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bae160b446ca4b27b3b2693d5141436d"
member_type: NON_VOTER
last_known_addr {
host: "127.11.109.60"
port: 43443
}
attrs {
promote: true
}
}
dest_uuid: "b547e792b86941398e04e3f6fb122765"
cas_config_opid_index: 5
I20260621 02:50:56.735960 17849 consensus_queue.cc:237] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } }
I20260621 02:50:56.737952 18408 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bae160b446ca4b27b3b2693d5141436d (127.11.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.742054 18457 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 LEARNER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 02:50:56.743851 18408 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.109.60:43443
I20260621 02:50:56.743796 18539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 0 FOLLOWER]: Advancing to term 1
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.1@KRBTEST.COM
I20260621 02:50:56.751514 18591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 02:50:56.758265 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.733703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.11.109.1:35299)
0621 02:50:56.734321 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.734330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.734359 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:50:56.734429 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.734437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.734713 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.734938 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.736898 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.738745 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.740788 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.741826 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.742663 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.748769 (+ 6106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.748787 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.748810 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.748863 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.753410 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.754343 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.754354 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.754360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.754562 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.755096 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.755104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.755110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.755637 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.757242 (+ 1605us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.757440 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.757703 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":305,"threads_started":1}
I20260621 02:50:56.759575 18033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:56.760015 18592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.760480 18592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:56.766724 18457 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bae160b446ca4b27b3b2693d5141436d (127.11.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.772886 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: NON_VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: true } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.773228 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:50:56.773772 18592 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: attempting to promote NON_VOTER d4aa724987c54feab6038e6ce99b67f5 to VOTER
I20260621 02:50:56.774525 18592 consensus_queue.cc:237] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } }
I20260621 02:50:56.776615 18457 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 LEARNER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 02:50:56.777158 18591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 02:50:56.777750 18539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 02:50:56.778259 18538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 02:50:56.778309 18411 master.cc:537] Master@127.11.109.60:43443 shutting down...
I20260621 02:50:56.780196 18592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d4aa724987c54feab6038e6ce99b67f5 (127.11.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.784614 18592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.784866 18592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:56.784174 18591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.001s
I20260621 02:50:56.785290 18408 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.785573 18408 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:56.785480 18457 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d4aa724987c54feab6038e6ce99b67f5 (127.11.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.789825 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:56.790138 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:50:56.794263 18411 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:56.794772 18411 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:56.795011 18411 tablet_replica.cc:330] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: stopping tablet replica
I20260621 02:50:56.802220 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.801827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.0.0.1:38524)
0621 02:50:56.801988 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.802094 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:50:56.802456 17836 consensus_peers.cc:487] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 -> Peer bae160b446ca4b27b3b2693d5141436d (127.11.109.60:43443): Couldn't send request to peer bae160b446ca4b27b3b2693d5141436d. Status: Network error: Client connection negotiation failed: client connection to 127.11.109.60:43443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 02:50:56.802611 18411 master.cc:552] Master@127.11.109.60:43443 shutdown complete.
I20260621 02:50:56.802706 18411 master_runner.cc:305] Clearing existing system tablet
I20260621 02:50:56.803086 18411 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:56.803207 18411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:56.806244 18411 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260621 02:50:56.807843 18598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:56.808223 18411 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:56.808471 18411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 02:50:56.808564 18411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
uuid: "bae160b446ca4b27b3b2693d5141436d"
format_stamp: "Formatted at 2026-06-21 02:50:56 on dist-test-slave-bc52"
server_key: "a6cc8f5055af17b842706aba4d88118e"
server_key_iv: "55d84011a9eb55303707ddebf5e78d0a"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.813532 18411 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:56.820303 18411 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 02:50:56.820465 18411 log.cc:1196] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 02:50:56.821010 18411 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Deleting consensus metadata
I20260621 02:50:56.821316 18411 master_runner.cc:315] Copying system tablet from 127.11.109.62:36053
I20260621 02:50:56.824105 18411 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: tablet copy: Beginning tablet copy session from remote peer at address 127.11.109.62:36053
I20260621 02:50:56.839553 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.824828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40280)
0621 02:50:56.825501 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:56.825525 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:56.825684 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:56.826058 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:56.826064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:56.826610 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:56.826885 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.826897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.828351 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.828357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.829933 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:56.829947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.830871 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.830877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:56.831054 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.831762 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:56.831785 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:56.832794 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:56.836682 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.836688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.836692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.837150 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.837572 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:56.837579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:56.837582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:56.837809 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:56.838435 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:56.838440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:56.838604 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 02:50:56.838757 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 02:50:56.839051 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":459,"threads_started":1}
I20260621 02:50:56.839570 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:56.824906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.11.109.62:36053)
0621 02:50:56.825137 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:56.825143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:56.825628 (+ 485us) server_negotiation.cc:407] Connection header received
0621 02:50:56.825813 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:56.825819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:56.825915 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:56.826031 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:56.827048 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.828193 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.830084 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:56.830718 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:56.830839 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:56.833030 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:56.833036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:56.833042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:56.833080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:56.836554 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.837320 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.837326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.837331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.837471 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:56.837918 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:56.837923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:56.837927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:56.838263 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:56.838482 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:56.838876 (+ 394us) server_negotiation.cc:299] Negotiation successful
0621 02:50:56.839159 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260621 02:50:56.840332 17869 tablet_copy_service.cc:140] P b547e792b86941398e04e3f6fb122765: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bae160b446ca4b27b3b2693d5141436d ({username='oryx', principal='oryx/127.11.109.60@KRBTEST.COM'} at 127.0.0.1:40280)
I20260621 02:50:56.840530 17869 tablet_copy_service.cc:161] P b547e792b86941398e04e3f6fb122765: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bae160b446ca4b27b3b2693d5141436d at {username='oryx', principal='oryx/127.11.109.60@KRBTEST.COM'} at 127.0.0.1:40280: session id = bae160b446ca4b27b3b2693d5141436d-00000000000000000000000000000000
I20260621 02:50:56.844219 17869 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 02:50:56.846027 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 02:50:56.853168 18411 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: tablet copy: Starting download of 0 data blocks...
I20260621 02:50:56.853462 18411 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: tablet copy: Starting download of 1 WAL segments...
I20260621 02:50:56.855759 18411 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 02:50:56.860148 17869 tablet_copy_service.cc:342] P b547e792b86941398e04e3f6fb122765: Request end of tablet copy session bae160b446ca4b27b3b2693d5141436d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.109.60@KRBTEST.COM'} at 127.0.0.1:40280
I20260621 02:50:56.860277 17869 tablet_copy_service.cc:433] P b547e792b86941398e04e3f6fb122765: ending tablet copy session bae160b446ca4b27b3b2693d5141436d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bae160b446ca4b27b3b2693d5141436d
I20260621 02:50:56.871598 18411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:56.874027 18605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:56.884584 18608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:56.885510 18606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:56.886621 18411 server_base.cc:1034] running on GCE node
I20260621 02:50:56.886952 18411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:56.887639 18411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:56.888837 18411 hybrid_clock.cc:648] HybridClock initialized: now 1782010256888779 us; error 55 us; skew 500 ppm
I20260621 02:50:56.889074 18411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:56 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010256, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:56.891597 18411 init.cc:377] Logged in from keytab as oryx/127.11.109.60@KRBTEST.COM (short username oryx)
I20260621 02:50:56.892927 18411 webserver.cc:460] Webserver started at http://127.11.109.60:44685/ using document root <none> and password file <none>
I20260621 02:50:56.893290 18411 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:56.893402 18411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:56.893554 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17910
I20260621 02:50:56.896397 18411 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260621 02:50:56.898257 18615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:56.898669 18411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260621 02:50:56.898823 18411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
uuid: "bae160b446ca4b27b3b2693d5141436d"
format_stamp: "Formatted at 2026-06-21 02:50:56 on dist-test-slave-bc52"
server_key: "a6cc8f5055af17b842706aba4d88118e"
server_key_iv: "55d84011a9eb55303707ddebf5e78d0a"
server_key_version: "encryptionkey@0"
I20260621 02:50:56.899094 18411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:56.912905 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:35299
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:36053
--webserver_port=33969
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:50:56.935127 18411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:56.939550 18411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:56.976450 18411 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.60:43443
I20260621 02:50:56.976533 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.60:43443 every 8 connection(s)
I20260621 02:50:56.977533 18411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
I20260621 02:50:56.982611 18668 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:50:56.986297 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Bootstrap starting.
I20260621 02:50:57.004817 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:50:57.005384 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: Bootstrap complete.
I20260621 02:50:57.006448 18668 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } }
I20260621 02:50:57.006735 18668 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bae160b446ca4b27b3b2693d5141436d, State: Initialized, Role: LEARNER
I20260621 02:50:57.007038 18668 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } }
I20260621 02:50:57.008004 18671 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:57.008441 18668 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:50:57.008460 18671 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: This master's current role is: LEARNER
I20260621 02:50:57.025876 18682 catalog_manager.cc:1259] Loaded cluster ID: 29e1e9b8c1bf476fbc48745f001454d0
I20260621 02:50:57.025992 18682 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: loading cluster ID for follower catalog manager: success
I20260621 02:50:57.028241 18682 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: acquiring CA information for follower catalog manager: success
I20260621 02:50:57.029963 18682 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:50:57.302114 18620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:57.302362 18620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:57.302521 18620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:57.310164 18620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:57.310254 18620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:57.310367 18620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:50:57.320591 18620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:35299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=33969
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:57.322006 18620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:57.323233 18620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:57.336478 18687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:57.336558 18688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:57.336534 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:57.337364 18620 server_base.cc:1034] running on GCE node
I20260621 02:50:57.337935 18620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:57.338874 18620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:57.340096 18620 hybrid_clock.cc:648] HybridClock initialized: now 1782010257340050 us; error 43 us; skew 500 ppm
I20260621 02:50:57.340404 18620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:57 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:57.343465 18620 init.cc:377] Logged in from keytab as oryx/127.11.109.1@KRBTEST.COM (short username oryx)
I20260621 02:50:57.344897 18620 webserver.cc:460] Webserver started at http://127.11.109.1:33969/ using document root <none> and password file <none>
I20260621 02:50:57.345598 18620 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:57.345744 18620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:57.351207 18620 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:50:57.353922 18697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:57.355248 18620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:50:57.355422 18620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "58ba662c9ec34515bc826593508a9c48"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "bc7f0700dabf5298101e8ca938fa0ef7"
server_key_iv: "8cda73eb761b09586518c170aba00992"
server_key_version: "encryptionkey@0"
I20260621 02:50:57.356362 18620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:57.378309 18620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:57.382125 18620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:57.382395 18620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:57.383289 18620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:57.384956 18620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:57.385072 18620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:57.385219 18620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:57.385306 18620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 02:50:57.396701 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.388241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.0.0.1:38538)
0621 02:50:57.388742 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:57.388774 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:57.388939 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:57.389875 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:57.389882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:57.389916 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:57.390069 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.390077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.391918 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.391924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.393650 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.393685 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.395801 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.395808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.396005 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.396054 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 02:50:57.396160 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:50:57.396328 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260621 02:50:57.396785 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.388385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.11.109.60:43443)
0621 02:50:57.389457 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:57.389463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:57.389481 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:50:57.389556 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:57.389563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:57.389700 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:57.389829 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:57.390216 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.391788 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.393837 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.395605 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.395779 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.396022 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:57.396202 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 02:50:57.396373 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":782,"threads_started":1}
I20260621 02:50:57.398182 18643 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260621 02:50:57.399197 18671 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b547e792b86941398e04e3f6fb122765. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:57.399544 18671 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: This master's current role is: LEARNER
I20260621 02:50:57.421403 18620 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:35299
I20260621 02:50:57.421484 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:35299 every 8 connection(s)
I20260621 02:50:57.422765 18620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:50:57.430063 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18620
I20260621 02:50:57.430416 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18044
Jun 21 02:50:57 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.60@KRBTEST.COM
Jun 21 02:50:57 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
Jun 21 02:50:57 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.2@KRBTEST.COM
Jun 21 02:50:57 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
I20260621 02:50:57.451655 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.426967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:38057 (local address 127.11.109.60:43443)
0621 02:50:57.427143 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:57.427149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:57.427267 (+ 118us) server_negotiation.cc:407] Connection header received
0621 02:50:57.430240 (+ 2973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:57.430246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:57.430328 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:57.430383 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:57.433807 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.434933 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.437618 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.438022 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.438160 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.444750 (+ 6590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:57.444755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:57.444758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:57.444789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:57.447897 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.448741 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.448748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.448755 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.448889 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.449420 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.449426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.449430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.449783 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:57.449938 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:57.450982 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0621 02:50:57.451216 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:50:57.453205 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.431927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.11.109.2:36383)
0621 02:50:57.432870 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:57.432880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:57.432905 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:50:57.432971 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:57.432981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:57.433230 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:57.433385 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:57.435255 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.436765 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.438700 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.439539 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.439696 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.446194 (+ 6498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:57.446204 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:57.446238 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:57.446290 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:57.449560 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.450857 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.450870 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.450872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.450990 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.451368 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.451373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.451375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.451999 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:57.452306 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:57.452556 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 02:50:57.452807 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":607,"threads_started":1}
I20260621 02:50:57.454107 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.426415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.11.109.1:38057)
0621 02:50:57.427115 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:57.427168 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:57.429931 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:57.430802 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:57.430814 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:57.431806 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:57.433666 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.433676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.435114 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.435126 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.437462 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.437471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.438306 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.438315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.438503 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.439750 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:57.439809 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:57.444501 (+ 4692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:57.448035 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:57.448043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:57.448061 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:57.448634 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:57.449029 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:57.449034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:57.449039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:57.449289 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:57.450090 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:57.450098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:57.450541 (+ 443us) client_negotiation.cc:769] Sending connection context
0621 02:50:57.451926 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0621 02:50:57.452504 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":213,"thread_start_us":270,"threads_started":1}
I20260621 02:50:57.454149 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.428273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.11.109.1:54777)
0621 02:50:57.428807 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:57.428826 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:57.429931 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:57.431167 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:57.431172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:57.432551 (+ 1379us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:57.433355 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.433368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.434974 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.434979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.438090 (+ 3111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.438107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.440006 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.440013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.440263 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.440818 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:57.440849 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:57.445744 (+ 4895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:57.450487 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:57.450492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:57.450496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:57.451006 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:57.452294 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:57.452298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:57.452301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:57.452450 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:57.453040 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:57.453047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:57.453234 (+ 187us) client_negotiation.cc:769] Sending connection context
0621 02:50:57.453356 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 02:50:57.453599 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1138,"spinlock_wait_cycles":21376,"thread_start_us":465,"threads_started":1}
I20260621 02:50:57.454914 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.427617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:50609)
0621 02:50:57.428239 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:57.428256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:57.429936 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:57.430802 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:57.430814 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:57.431765 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:57.432798 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.432826 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.435003 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.435009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.437432 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:57.437447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.439150 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.439157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:57.439322 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.439958 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:57.439985 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:57.443684 (+ 3699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:57.451114 (+ 7430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:57.451118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:57.451122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:57.451561 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:57.451994 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:57.451998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:57.452002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:57.452162 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:57.454047 (+ 1885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:57.454053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:57.454222 (+ 169us) client_negotiation.cc:769] Sending connection context
0621 02:50:57.454368 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 02:50:57.454588 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":398,"thread_start_us":281,"threads_started":1}
I20260621 02:50:57.455137 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.427753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:50609 (local address 127.11.109.62:36053)
0621 02:50:57.429799 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:57.429805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:57.429827 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:50:57.430348 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:57.430353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:57.430438 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:57.430617 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:57.433381 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.434611 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.438262 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.438986 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.439112 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.443907 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:57.443912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:57.443915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:57.443946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:57.450730 (+ 6784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.451703 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.451708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.451711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.451904 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.452532 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.452537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.452541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.453533 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:57.453689 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:57.454514 (+ 825us) server_negotiation.cc:299] Negotiation successful
0621 02:50:57.454760 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1904,"thread_start_us":1781,"threads_started":1}
I20260621 02:50:57.455261 18167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:57.455826 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:57.428632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:54777 (local address 127.11.109.61:42527)
0621 02:50:57.430902 (+ 2270us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:57.430906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:57.430924 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:50:57.430967 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:57.430970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:57.431028 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:57.431137 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:57.433829 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.434838 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.438728 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:57.439584 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:57.439729 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:57.445960 (+ 6231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:57.445967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:57.445971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:57.446011 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:57.450340 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.451735 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.451739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.451743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.451875 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:57.452586 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:57.452590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:57.452594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:57.452906 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:57.455139 (+ 2233us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:57.455257 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 02:50:57.455486 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":715,"threads_started":1}
I20260621 02:50:57.456316 18813 heartbeater.cc:347] Connected to a master server at 127.11.109.60:43443
I20260621 02:50:57.456617 18813 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:57.457283 18815 heartbeater.cc:347] Connected to a master server at 127.11.109.61:42527
I20260621 02:50:57.457460 18815 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:57.457592 18813 heartbeater.cc:511] Master 127.11.109.60:43443 requested a full tablet report, sending...
I20260621 02:50:57.458076 18815 heartbeater.cc:511] Master 127.11.109.61:42527 requested a full tablet report, sending...
I20260621 02:50:57.458389 18814 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36053
I20260621 02:50:57.458508 18814 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:57.458912 18814 heartbeater.cc:511] Master 127.11.109.62:36053 requested a full tablet report, sending...
I20260621 02:50:57.459904 18447 ts_manager.cc:194] Registered new tserver with Master: 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:50:57.460290 18633 ts_manager.cc:194] Registered new tserver with Master: 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:50:57.461267 17849 ts_manager.cc:194] Re-registered known tserver with Master: 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:50:57.462013 17849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:50609
I20260621 02:50:57.546267 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18044
I20260621 02:50:57.562081 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:36383
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:36053
--webserver_port=44403
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:50:57.897547 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:57.897821 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:57.897974 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:57.905457 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:57.905588 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260621 02:50:57.905023 18829 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: NON_VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: true } } }
I20260621 02:50:57.905674 18829 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
W20260621 02:50:57.905807 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:50:57.906497 18831 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: attempting to promote NON_VOTER bae160b446ca4b27b3b2693d5141436d to VOTER
I20260621 02:50:57.907022 18831 consensus_queue.cc:237] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:50:57.909250 18643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 LEARNER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 02:50:57.909369 18457 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Refusing update from remote peer b547e792b86941398e04e3f6fb122765: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 02:50:57.909885 18830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 02:50:57.910346 18831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 02:50:57.913015 18831 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bae160b446ca4b27b3b2693d5141436d (127.11.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.913692 18457 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bae160b446ca4b27b3b2693d5141436d (127.11.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.916653 18831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.916929 18831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:57.917234 18829 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.917440 18829 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:57.917938 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:36383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=44403
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:57.918932 18831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.919188 18831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765 [sys.catalog]: This master's current role is: LEADER
I20260621 02:50:57.919607 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:57.919502 18643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bae160b446ca4b27b3b2693d5141436d (127.11.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.921260 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:50:57.923722 18836 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.924077 18836 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:50:57.924213 18837 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b547e792b86941398e04e3f6fb122765" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:50:57.924520 18837 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: This master's current role is: FOLLOWER
W20260621 02:50:57.934128 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:57.934166 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:57.934201 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:57.938723 18825 server_base.cc:1034] running on GCE node
I20260621 02:50:57.939364 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:57.940354 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:57.941553 18825 hybrid_clock.cc:648] HybridClock initialized: now 1782010257941477 us; error 71 us; skew 500 ppm
I20260621 02:50:57.941859 18825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:57 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:57.944833 18825 init.cc:377] Logged in from keytab as oryx/127.11.109.2@KRBTEST.COM (short username oryx)
I20260621 02:50:57.946346 18825 webserver.cc:460] Webserver started at http://127.11.109.2:44403/ using document root <none> and password file <none>
I20260621 02:50:57.946995 18825 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:57.947099 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:57.952135 18825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:50:57.954829 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:57.956238 18825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:50:57.956406 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea"
format_stamp: "Formatted at 2026-06-21 02:50:54 on dist-test-slave-bc52"
server_key: "61b1604bfed0596e7375168eb33bc388"
server_key_iv: "4bd6157e4a4599eeebe8921693de6ff7"
server_key_version: "encryptionkey@0"
I20260621 02:50:57.957358 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:57.988413 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:57.992133 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:57.992409 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:57.993388 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:57.994624 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:57.994740 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:57.994859 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:57.994971 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:58.027556 18825 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:36383
I20260621 02:50:58.027638 18961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:36383 every 8 connection(s)
I20260621 02:50:58.029062 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:50:58.030213 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18825
I20260621 02:50:58.030491 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18177
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.3@KRBTEST.COM
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010257, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.60@KRBTEST.COM
I20260621 02:50:58.063975 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.033604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:51971 (local address 127.11.109.62:36053)
0621 02:50:58.034697 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.034707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.036961 (+ 2254us) server_negotiation.cc:407] Connection header received
0621 02:50:58.041542 (+ 4581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.041551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.041668 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.041788 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.044378 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.045716 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.049102 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.049982 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.050115 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.055890 (+ 5775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.055898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.055904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.055957 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.059639 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.060626 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.060634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.060637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.060781 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.061357 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.061362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.061366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.061760 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.062483 (+ 723us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.063317 (+ 834us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.063551 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":818,"threads_started":1}
I20260621 02:50:58.065032 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.036577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.11.109.2:40291)
0621 02:50:58.045855 (+ 9278us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.045873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.046032 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.046640 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.046645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.047781 (+ 1136us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:58.048155 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.048169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.049577 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.049581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.051179 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.051193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.052217 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.052223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.052400 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.053085 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:58.053105 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:58.056214 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:58.060289 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.060293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.060298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.060854 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.061366 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.061371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.061377 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.061562 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.062206 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:58.062216 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:58.062702 (+ 486us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.063031 (+ 329us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.063531 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9144,"mutex_wait_us":4,"thread_start_us":8984,"threads_started":1}
I20260621 02:50:58.065040 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.032810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:51971)
0621 02:50:58.035359 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.035405 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.041155 (+ 5750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.042372 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.042386 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.043219 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:58.044121 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.044149 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.045888 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.045893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.048912 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.048930 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.050442 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.050451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.050835 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.052402 (+ 1567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:58.052426 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:58.055582 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:58.059833 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.059843 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.059861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.060528 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.060905 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.060910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.060914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.061217 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.062590 (+ 1373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:58.062597 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:58.062771 (+ 174us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.063020 (+ 249us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.064464 (+ 1444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"mutex_wait_us":1044,"spinlock_wait_cycles":21376,"thread_start_us":2004,"threads_started":1}
I20260621 02:50:58.066262 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.036994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:40291 (local address 127.11.109.61:42527)
0621 02:50:58.045445 (+ 8451us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.045451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.046142 (+ 691us) server_negotiation.cc:407] Connection header received
0621 02:50:58.046220 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.046227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.046313 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.046412 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.048322 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.049426 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.051316 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.052047 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.052174 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.056405 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.056411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.056415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.056449 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.060112 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.061057 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.061064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.061067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.061238 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.061714 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.061720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.061724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.062055 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.065482 (+ 3427us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.065620 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.065858 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8255,"thread_start_us":8105,"threads_started":1}
I20260621 02:50:58.066849 18963 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36053
I20260621 02:50:58.067054 18964 heartbeater.cc:347] Connected to a master server at 127.11.109.61:42527
I20260621 02:50:58.067216 18963 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:58.067221 18964 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:58.068114 18963 heartbeater.cc:511] Master 127.11.109.62:36053 requested a full tablet report, sending...
I20260621 02:50:58.068437 18964 heartbeater.cc:511] Master 127.11.109.61:42527 requested a full tablet report, sending...
I20260621 02:50:58.068892 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.031553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.11.109.3:35659)
0621 02:50:58.044013 (+ 12460us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.044036 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.044752 (+ 716us) server_negotiation.cc:407] Connection header received
0621 02:50:58.044973 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.044984 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.045259 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.045417 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.048239 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.049734 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.051922 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.052776 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.052891 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.059312 (+ 6421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.059325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.059342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.059386 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.064454 (+ 5068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.065874 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.065881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.065884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.066048 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.066463 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.066468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.066471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.066898 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.067183 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.068255 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.068495 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12230,"thread_start_us":12474,"threads_started":1}
I20260621 02:50:58.069800 17849 ts_manager.cc:194] Re-registered known tserver with Master: e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383)
I20260621 02:50:58.069872 18447 ts_manager.cc:194] Registered new tserver with Master: e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383)
I20260621 02:50:58.070369 18300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:58.070940 17849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:51971
I20260621 02:50:58.071545 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.048039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.11.109.2:40361)
0621 02:50:58.051698 (+ 3659us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.051715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.051812 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.054076 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.054086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.055180 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:58.055553 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.055565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.057051 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.057058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.058689 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.058704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.059938 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.059944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.060113 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.060973 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:58.060994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:58.063768 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:58.067894 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.067900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.067903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.069013 (+ 1110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.069469 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.069474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.069477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.069634 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.070291 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:58.070298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:58.070440 (+ 142us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.070602 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.070981 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3537,"thread_start_us":3389,"threads_started":1}
I20260621 02:50:58.072681 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.046050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:40361 (local address 127.11.109.60:43443)
0621 02:50:58.052912 (+ 6862us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.052917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.052933 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:50:58.053670 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.053675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.053759 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.053901 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.055761 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.056885 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.058855 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.059553 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.059676 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.063997 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.064003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.064008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.064047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.067773 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.069127 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.069132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.069135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.069335 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.069754 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.069758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.069761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.070069 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.070199 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.071870 (+ 1671us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.072133 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6687,"thread_start_us":5934,"threads_started":1}
I20260621 02:50:58.073450 18962 heartbeater.cc:347] Connected to a master server at 127.11.109.60:43443
I20260621 02:50:58.073617 18962 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:58.074047 18962 heartbeater.cc:511] Master 127.11.109.60:43443 requested a full tablet report, sending...
I20260621 02:50:58.075325 18633 ts_manager.cc:194] Registered new tserver with Master: e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383)
I20260621 02:50:58.154228 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18177
I20260621 02:50:58.170327 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:35659
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:36053
--webserver_port=45577
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:50:58.464192 18814 heartbeater.cc:503] Master 127.11.109.62:36053 was elected leader, sending a full tablet report...
W20260621 02:50:58.491060 18975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:50:58.491303 18975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:50:58.491465 18975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:50:58.498482 18975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:50:58.498574 18975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:50:58.498756 18975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:50:58.508368 18975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:35659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=45577
--tserver_master_addrs=127.11.109.62:36053,127.11.109.61:42527,127.11.109.60:43443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:50:58.509817 18975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:50:58.511183 18975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:50:58.523769 18983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:58.523801 18981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:50:58.524091 18980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:50:58.524901 18975 server_base.cc:1034] running on GCE node
I20260621 02:50:58.525537 18975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:50:58.526425 18975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:50:58.527606 18975 hybrid_clock.cc:648] HybridClock initialized: now 1782010258527562 us; error 42 us; skew 500 ppm
I20260621 02:50:58.527896 18975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010258, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:50:58.531037 18975 init.cc:377] Logged in from keytab as oryx/127.11.109.3@KRBTEST.COM (short username oryx)
I20260621 02:50:58.532416 18975 webserver.cc:460] Webserver started at http://127.11.109.3:45577/ using document root <none> and password file <none>
I20260621 02:50:58.533044 18975 fs_manager.cc:359] Metadata directory not provided
I20260621 02:50:58.533215 18975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:50:58.538383 18975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260621 02:50:58.540933 18990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:58.542227 18975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:50:58.542384 18975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "2a15586458ed42b2ae80e2dbc4b6d669"
format_stamp: "Formatted at 2026-06-21 02:50:55 on dist-test-slave-bc52"
server_key: "24e0658ab253f9b55e6e9338bdaee528"
server_key_iv: "172418a63810564e4f9dbcdfe2d8d0de"
server_key_version: "encryptionkey@0"
I20260621 02:50:58.543296 18975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:50:58.557251 18975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:50:58.560964 18975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:50:58.561321 18975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:50:58.562271 18975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:50:58.563498 18975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:50:58.563580 18975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:58.563719 18975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:50:58.563813 18975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:50:58.596231 18975 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35659
I20260621 02:50:58.596342 19102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35659 every 8 connection(s)
I20260621 02:50:58.597663 18975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:50:58.598835 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 18975
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010258, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:50:58.641355 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.601423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:60011)
0621 02:50:58.612810 (+ 11387us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.612876 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.616319 (+ 3443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.617788 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.617800 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.618636 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:58.619598 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.619628 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.621777 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.621784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.624302 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.624321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.625806 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.625815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.626215 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.627098 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:58.627137 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:58.630665 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:58.635876 (+ 5211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.635888 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.635907 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.636639 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.637231 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.637239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.637245 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.637607 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.638337 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:58.638348 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:58.638810 (+ 462us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.639120 (+ 310us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.639706 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10979,"thread_start_us":11666,"threads_started":1}
I20260621 02:50:58.643168 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.600597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:60011 (local address 127.11.109.62:36053)
0621 02:50:58.610512 (+ 9915us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.610518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.613051 (+ 2533us) server_negotiation.cc:407] Connection header received
0621 02:50:58.617257 (+ 4206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.617264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.617356 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.619870 (+ 2514us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.620076 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.621575 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.624542 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.625504 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.626713 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.630975 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.630984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.630992 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.631055 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.635677 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.636875 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.636884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.636890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.637076 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.637779 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.637787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.637793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.638181 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.642397 (+ 4216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.642541 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.642772 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9800,"thread_start_us":9885,"threads_started":1}
I20260621 02:50:58.648643 19104 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36053
I20260621 02:50:58.648914 19104 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:58.649849 19104 heartbeater.cc:511] Master 127.11.109.62:36053 requested a full tablet report, sending...
I20260621 02:50:58.653626 17849 ts_manager.cc:194] Re-registered known tserver with Master: 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659)
I20260621 02:50:58.654785 17849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:60011
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010258, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.61@KRBTEST.COM
Jun 21 02:50:58 dist-test-slave-bc52 krb5kdc[17804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010258, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.60@KRBTEST.COM
I20260621 02:50:58.668042 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.643164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.11.109.60:43443)
0621 02:50:58.648601 (+ 5437us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.648609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.648629 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:50:58.648695 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.648699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.648768 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.651116 (+ 2348us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.651506 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.652672 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.654514 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.655193 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.655306 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.659808 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.659816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.659820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.659855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.664206 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.664937 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.664944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.664950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.665132 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.665606 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.665612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.665618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.665993 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.666276 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.667409 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.667619 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5298,"thread_start_us":4271,"threads_started":1}
I20260621 02:50:58.669340 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.618661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.11.109.62:36053)
0621 02:50:58.644459 (+ 25798us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.644467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.644488 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:50:58.644542 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.644547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.644640 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.644765 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.649482 (+ 4717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.650563 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.653277 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.653895 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.655395 (+ 1500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.660978 (+ 5583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.660985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.660990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.661050 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.664888 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.666553 (+ 1665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.666561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.666565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.666716 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.667219 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.667224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.667227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.667555 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.667757 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.668694 (+ 937us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.668919 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1364,"server-negotiator.queue_time_us":25642,"thread_start_us":26246,"threads_started":1}
I20260621 02:50:58.669802 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.629966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.11.109.3:45741)
0621 02:50:58.642116 (+ 12150us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.642137 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.642239 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.649371 (+ 7132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.649377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.649884 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:58.650796 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.650817 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.653446 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.653451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.655417 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.655431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.656367 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.656372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.656513 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.657097 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:58.657117 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:58.659845 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:58.667025 (+ 7180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.667032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.667038 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.667532 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.667989 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.667993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.667997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.668149 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.668774 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:58.668779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:58.668923 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.669065 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.669370 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12006,"thread_start_us":15290,"threads_started":1}
I20260621 02:50:58.670740 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.629946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:45741 (local address 127.11.109.60:43443)
0621 02:50:58.647651 (+ 17705us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.647658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.647677 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:50:58.647737 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.647743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.647831 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.650548 (+ 2717us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.652067 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.653303 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.655588 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.656225 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.656353 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.660161 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.660171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.660178 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.660231 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.666886 (+ 6655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.667694 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.667699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.667702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.667859 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.668289 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.668293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.668296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.668634 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.670046 (+ 1412us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.670172 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.670387 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17575,"thread_start_us":19243,"threads_started":1}
I20260621 02:50:58.671604 19103 heartbeater.cc:347] Connected to a master server at 127.11.109.60:43443
I20260621 02:50:58.671758 19103 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:58.672163 19103 heartbeater.cc:511] Master 127.11.109.60:43443 requested a full tablet report, sending...
I20260621 02:50:58.672787 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.614153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.11.109.3:49023)
0621 02:50:58.643610 (+ 29457us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.643633 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.643831 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.644612 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.644619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.645460 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:50:58.645822 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.645836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.647713 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.647720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.649322 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.649334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.651776 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.651783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.654016 (+ 2233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.654852 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:50:58.654874 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:50:58.661453 (+ 6579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:50:58.669011 (+ 7558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.669015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.669018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.669626 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.670376 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:50:58.670383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:50:58.670386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:50:58.670533 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:50:58.671760 (+ 1227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:50:58.671767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:50:58.671915 (+ 148us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.672077 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.672379 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15375,"thread_start_us":15216,"threads_started":1}
I20260621 02:50:58.673362 18633 ts_manager.cc:194] Registered new tserver with Master: 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659)
I20260621 02:50:58.674041 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.622811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.11.109.61:42527)
0621 02:50:58.643475 (+ 20664us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.643485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.643511 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:50:58.643584 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.643592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.643690 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.643832 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.652018 (+ 8186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.653128 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.659764 (+ 6636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.660481 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.660629 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.665261 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.665268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.665273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.665310 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.668573 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.670400 (+ 1827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.670405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.670407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.670552 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.670961 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.670966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.670970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.671284 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.671537 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.672741 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.673159 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":20482,"thread_start_us":21155,"threads_started":1}
I20260621 02:50:58.674556 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.614254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:49023 (local address 127.11.109.61:42527)
0621 02:50:58.629241 (+ 14987us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.629249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.643723 (+ 14474us) server_negotiation.cc:407] Connection header received
0621 02:50:58.644138 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.644146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.644235 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.644323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:58.646208 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.647547 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.650746 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.651412 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.651505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.661675 (+ 10170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:58.661683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:58.661689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:58.661733 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:58.668877 (+ 7144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.670076 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.670083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.670087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.670221 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:58.670663 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:58.670667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:58.670673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:58.670986 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:58.671131 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.672344 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.674164 (+ 1820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14839,"thread_start_us":14759,"threads_started":1}
I20260621 02:50:58.675247 19105 heartbeater.cc:347] Connected to a master server at 127.11.109.61:42527
I20260621 02:50:58.675367 19105 heartbeater.cc:464] Registering TS with master...
I20260621 02:50:58.675643 19105 heartbeater.cc:511] Master 127.11.109.61:42527 requested a full tablet report, sending...
I20260621 02:50:58.676724 18447 ts_manager.cc:194] Registered new tserver with Master: 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659)
I20260621 02:50:58.674453 17849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:50:58.682279 17849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:50:58.707284 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.698509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36383 (local address 127.0.0.1:47146)
0621 02:50:58.699158 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.699178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.699301 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.700576 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.700581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.700616 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:58.700919 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.700926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.703311 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.703316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.704853 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.704865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.706459 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.706465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.706656 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.706695 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.706765 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.706905 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":384,"threads_started":1}
I20260621 02:50:58.709460 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.698759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.11.109.2:36383)
0621 02:50:58.699911 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.699920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.699950 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:50:58.700063 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.700072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.700352 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.701100 (+ 748us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:58.701777 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.703152 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.705011 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.706247 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.708256 (+ 2009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.708538 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.708613 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.708766 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":724,"threads_started":1}
I20260621 02:50:58.712615 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.699203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35659 (local address 127.0.0.1:40524)
0621 02:50:58.701648 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.701666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.701781 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.702357 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.702364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.702394 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:58.703146 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.703164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.705876 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.705881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.708345 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.708377 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.711421 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.711430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.711687 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.711736 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.711828 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.712041 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2298,"mutex_wait_us":1,"thread_start_us":2191,"threads_started":1}
I20260621 02:50:58.714231 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.699507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.11.109.3:35659)
0621 02:50:58.700241 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.700252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.701759 (+ 1507us) server_negotiation.cc:407] Connection header received
0621 02:50:58.701919 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.701927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.702178 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.702367 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:58.703406 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.705717 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.708598 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.710499 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.713314 (+ 2815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.713525 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.713575 (+ 50us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.713679 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":816,"threads_started":1}
I20260621 02:50:58.714072 18897 tablet_service.cc:1467] Processing CreateTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca (DEFAULT_TABLE table=test-table [id=19109218459c49d996c1d5f02345e385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:58.715516 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bcde4a41fd4096afbe97ae6e7a51ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:58.715705 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.702238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35299 (local address 127.0.0.1:54258)
0621 02:50:58.707527 (+ 5289us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.707543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.707677 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.709045 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.709049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.709073 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:58.709264 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.709275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.710621 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.710627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.712452 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.712499 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.714913 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.714919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.715098 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.715130 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.715236 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.715359 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5164,"thread_start_us":10195,"threads_started":1}
I20260621 02:50:58.715735 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.701965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.11.109.1:35299)
0621 02:50:58.705385 (+ 3420us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.705395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.708521 (+ 3126us) server_negotiation.cc:407] Connection header received
0621 02:50:58.708624 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.708632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.708868 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.709049 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:58.709389 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.710489 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.712881 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.714746 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.714890 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.715069 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.715283 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.715439 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":399,"threads_started":1}
I20260621 02:50:58.717353 19038 tablet_service.cc:1467] Processing CreateTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca (DEFAULT_TABLE table=test-table [id=19109218459c49d996c1d5f02345e385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:58.718679 19038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bcde4a41fd4096afbe97ae6e7a51ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:58.719282 18748 tablet_service.cc:1467] Processing CreateTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca (DEFAULT_TABLE table=test-table [id=19109218459c49d996c1d5f02345e385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:50:58.720562 18748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bcde4a41fd4096afbe97ae6e7a51ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:50:58.727787 19137 tablet_bootstrap.cc:492] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Bootstrap starting.
I20260621 02:50:58.730931 19137 tablet_bootstrap.cc:654] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:58.731869 19137 log.cc:824] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:58.735365 19137 tablet_bootstrap.cc:492] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: No bootstrap required, opened a new log
I20260621 02:50:58.735617 19137 ts_tablet_manager.cc:1397] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:50:58.736434 19140 tablet_bootstrap.cc:492] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Bootstrap starting.
I20260621 02:50:58.739383 19140 tablet_bootstrap.cc:654] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:58.740329 19140 log.cc:824] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:58.741314 19139 tablet_bootstrap.cc:492] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Bootstrap starting.
I20260621 02:50:58.743700 19140 tablet_bootstrap.cc:492] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: No bootstrap required, opened a new log
I20260621 02:50:58.743602 19137 raft_consensus.cc:348] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.743961 19140 ts_tablet_manager.cc:1397] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260621 02:50:58.743993 19137 raft_consensus.cc:374] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:58.744097 19137 raft_consensus.cc:729] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76cd48a4a9a48f6ac96f2bee324c7ea, State: Initialized, Role: FOLLOWER
I20260621 02:50:58.744470 19139 tablet_bootstrap.cc:654] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Neither blocks nor log segments found. Creating new log.
I20260621 02:50:58.744714 19137 consensus_queue.cc:260] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.745615 19139 log.cc:824] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Log is configured to *not* fsync() on all Append() calls
I20260621 02:50:58.748042 19137 ts_tablet_manager.cc:1428] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260621 02:50:58.748319 18963 heartbeater.cc:503] Master 127.11.109.62:36053 was elected leader, sending a full tablet report...
I20260621 02:50:58.755815 19139 tablet_bootstrap.cc:492] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: No bootstrap required, opened a new log
I20260621 02:50:58.756134 19139 ts_tablet_manager.cc:1397] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260621 02:50:58.756213 19140 raft_consensus.cc:348] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.756633 19140 raft_consensus.cc:374] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:58.756759 19140 raft_consensus.cc:729] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ba662c9ec34515bc826593508a9c48, State: Initialized, Role: FOLLOWER
I20260621 02:50:58.757377 19140 consensus_queue.cc:260] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.759017 19140 ts_tablet_manager.cc:1428] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260621 02:50:58.765295 19139 raft_consensus.cc:348] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.765609 19139 raft_consensus.cc:374] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:50:58.765678 19139 raft_consensus.cc:729] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a15586458ed42b2ae80e2dbc4b6d669, State: Initialized, Role: FOLLOWER
I20260621 02:50:58.766114 19139 consensus_queue.cc:260] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.766979 19104 heartbeater.cc:503] Master 127.11.109.62:36053 was elected leader, sending a full tablet report...
I20260621 02:50:58.767408 19139 ts_tablet_manager.cc:1428] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260621 02:50:58.789595 18970 tablet.cc:2369] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:50:58.856010 19106 tablet.cc:2369] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:50:58.893810 19142 raft_consensus.cc:482] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:50:58.894105 19142 raft_consensus.cc:504] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.895794 19142 leader_election.cc:288] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659), 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:50:58.903052 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.896080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:59619 (local address 127.11.109.3:35659)
0621 02:50:58.896336 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.896346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.896726 (+ 380us) server_negotiation.cc:407] Connection header received
0621 02:50:58.896919 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.896925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.897014 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.897117 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:58.897794 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.898674 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.901229 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.902260 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.902381 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.902429 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.902793 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.902875 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:50:58.903493 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.895999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35659 (local address 127.11.109.2:59619)
0621 02:50:58.896610 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.896630 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.896781 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.897242 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.897246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.897277 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:58.897656 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.897669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.898806 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.898812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.901049 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.901060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.902441 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.902447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.902631 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.902668 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.902776 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.902906 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1,"thread_start_us":309,"threads_started":1}
I20260621 02:50:58.904055 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.896995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:36587 (local address 127.11.109.1:35299)
0621 02:50:58.897246 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.897252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.897431 (+ 179us) server_negotiation.cc:407] Connection header received
0621 02:50:58.897564 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.897569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.897652 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.897754 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:50:58.898282 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.899513 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.901966 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.902995 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.903166 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.903242 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.903746 (+ 504us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.903849 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:50:58.904664 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.896647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35299 (local address 127.11.109.2:36587)
0621 02:50:58.897327 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:58.897355 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:50:58.897456 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:50:58.897782 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:50:58.897786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:50:58.897807 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:50:58.898115 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.898125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.899645 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.899649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.901769 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:50:58.901782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.903400 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.903405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:50:58.903529 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.903578 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 02:50:58.904231 (+ 653us) client_negotiation.cc:240] Negotiation successful
0621 02:50:58.904360 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":340,"threads_started":1}
I20260621 02:50:58.904471 19058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bcde4a41fd4096afbe97ae6e7a51ca" candidate_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" is_pre_election: true
I20260621 02:50:58.905009 19058 raft_consensus.cc:2454] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76cd48a4a9a48f6ac96f2bee324c7ea in term 0.
I20260621 02:50:58.905663 18853 leader_election.cc:302] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a15586458ed42b2ae80e2dbc4b6d669, e76cd48a4a9a48f6ac96f2bee324c7ea; no voters:
I20260621 02:50:58.905608 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bcde4a41fd4096afbe97ae6e7a51ca" candidate_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ba662c9ec34515bc826593508a9c48" is_pre_election: true
I20260621 02:50:58.906025 19142 raft_consensus.cc:2781] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:50:58.906037 18768 raft_consensus.cc:2454] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76cd48a4a9a48f6ac96f2bee324c7ea in term 0.
I20260621 02:50:58.906129 19142 raft_consensus.cc:482] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:50:58.906258 19142 raft_consensus.cc:3037] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:58.908725 19142 raft_consensus.cc:504] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.909435 19142 leader_election.cc:288] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [CANDIDATE]: Term 1 election: Requested vote from peers 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659), 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:50:58.909811 19058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bcde4a41fd4096afbe97ae6e7a51ca" candidate_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a15586458ed42b2ae80e2dbc4b6d669"
I20260621 02:50:58.909868 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bcde4a41fd4096afbe97ae6e7a51ca" candidate_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ba662c9ec34515bc826593508a9c48"
I20260621 02:50:58.909972 19058 raft_consensus.cc:3037] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:58.910080 18768 raft_consensus.cc:3037] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:50:58.912267 18768 raft_consensus.cc:2454] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76cd48a4a9a48f6ac96f2bee324c7ea in term 1.
I20260621 02:50:58.912621 18853 leader_election.cc:302] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58ba662c9ec34515bc826593508a9c48, e76cd48a4a9a48f6ac96f2bee324c7ea; no voters:
I20260621 02:50:58.912614 19058 raft_consensus.cc:2454] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76cd48a4a9a48f6ac96f2bee324c7ea in term 1.
I20260621 02:50:58.912853 19142 raft_consensus.cc:2781] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:50:58.913357 19142 raft_consensus.cc:686] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 LEADER]: Becoming Leader. State: Replica: e76cd48a4a9a48f6ac96f2bee324c7ea, State: Running, Role: LEADER
I20260621 02:50:58.913808 19142 consensus_queue.cc:237] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:50:58.918833 17849 catalog_manager.cc:5495] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea reported cstate change: term changed from 0 to 1, leader changed from <none> to e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2). New cstate: current_term: 1 leader_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } health_report { overall_health: UNKNOWN } } }
W20260621 02:50:58.926908 18816 tablet.cc:2369] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:50:58.952574 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:58.948026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.11.109.2:36383)
0621 02:50:58.948182 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:58.948187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:58.948275 (+ 88us) server_negotiation.cc:407] Connection header received
0621 02:50:58.948442 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:58.948445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:58.948513 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:58.948641 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:58.948915 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.949613 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.950822 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:58.951181 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:58.951269 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:58.951635 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:58.952154 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:58.952257 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:58.952325 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 02:50:58.952392 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:50:58.960450 18768 raft_consensus.cc:1261] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Refusing update from remote peer e76cd48a4a9a48f6ac96f2bee324c7ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:58.960546 19058 raft_consensus.cc:1261] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Refusing update from remote peer e76cd48a4a9a48f6ac96f2bee324c7ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:50:58.961483 19148 consensus_queue.cc:1035] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:58.961889 19142 consensus_queue.cc:1035] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:50:58.989295 19153 mvcc.cc:205] Tried to move back new op lower bound from 7299114020691775488 to 7299114020514439168. Current Snapshot: MvccSnapshot[applied={T|T < 7299114020691775488}]
I20260621 02:50:58.992532 17849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:58.992831 17849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:50:59.003077 17849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b547e792b86941398e04e3f6fb122765: Sending DeleteTablet for 3 replicas of tablet 92bcde4a41fd4096afbe97ae6e7a51ca
I20260621 02:50:59.004374 18897 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:58 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:47146
I20260621 02:50:59.005650 19038 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:58 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:40524
I20260621 02:50:59.005609 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:58 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:54258
I20260621 02:50:59.006208 18748 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bcde4a41fd4096afbe97ae6e7a51ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:50:58 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:54258
I20260621 02:50:59.014169 19162 tablet_replica.cc:330] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: stopping tablet replica
I20260621 02:50:59.014709 19162 raft_consensus.cc:2229] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:59.015300 19162 raft_consensus.cc:2258] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:59.016871 19161 tablet_replica.cc:330] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: stopping tablet replica
I20260621 02:50:59.017361 19161 raft_consensus.cc:2229] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:50:59.017699 19162 ts_tablet_manager.cc:1905] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:59.018163 19161 raft_consensus.cc:2258] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:59.020726 19161 ts_tablet_manager.cc:1905] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:59.022775 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 17817
I20260621 02:50:59.023864 17879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:59.033493 19160 tablet_replica.cc:330] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: stopping tablet replica
I20260621 02:50:59.038872 19160 raft_consensus.cc:2229] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:50:59.039574 19162 ts_tablet_manager.cc:1918] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:59.039588 19160 raft_consensus.cc:2258] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:50:59.039793 19162 log.cc:1196] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/92bcde4a41fd4096afbe97ae6e7a51ca
I20260621 02:50:59.041700 19162 ts_tablet_manager.cc:1939] T 92bcde4a41fd4096afbe97ae6e7a51ca P 58ba662c9ec34515bc826593508a9c48: Deleting consensus metadata
I20260621 02:50:59.048523 19160 ts_tablet_manager.cc:1905] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:50:59.050554 19161 ts_tablet_manager.cc:1918] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:59.051431 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:59.008933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.11.109.61:42527)
0621 02:50:59.009143 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:59.009150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:59.018640 (+ 9490us) server_negotiation.cc:407] Connection header received
0621 02:50:59.019244 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:59.019252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:59.019351 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:59.019429 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:59.025638 (+ 6209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.026790 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.031194 (+ 4404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.031844 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.031945 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:59.041967 (+ 10022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:59.041974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:59.041979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:59.042020 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:59.045523 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.047490 (+ 1967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.047498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.047501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.047653 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.049051 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.049056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.049060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.049436 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:59.049654 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:59.050882 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0621 02:50:59.051111 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:50:59.060580 19160 ts_tablet_manager.cc:1918] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:50:59.060827 19161 log.cc:1196] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/92bcde4a41fd4096afbe97ae6e7a51ca
I20260621 02:50:59.061487 19161 ts_tablet_manager.cc:1939] T 92bcde4a41fd4096afbe97ae6e7a51ca P 2a15586458ed42b2ae80e2dbc4b6d669: Deleting consensus metadata
I20260621 02:50:59.063392 19160 log.cc:1196] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/92bcde4a41fd4096afbe97ae6e7a51ca
I20260621 02:50:59.064074 19160 ts_tablet_manager.cc:1939] T 92bcde4a41fd4096afbe97ae6e7a51ca P e76cd48a4a9a48f6ac96f2bee324c7ea: Deleting consensus metadata
I20260621 02:50:59.080344 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:59.021877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.11.109.60:43443)
0621 02:50:59.022082 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:59.022089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:59.052674 (+ 30585us) server_negotiation.cc:407] Connection header received
0621 02:50:59.052741 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:59.052746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:59.052839 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:59.052931 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:59.066626 (+ 13695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.068187 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.070081 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.070728 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.070856 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:59.074194 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:59.074201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:59.074206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:59.074246 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:59.077626 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.078352 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.078358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.078362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.078499 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.078908 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.078913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.078917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.079234 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:59.079468 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:59.079797 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 02:50:59.080000 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:50:59.170728 17836 catalog_manager.cc:4841] TS 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659): tablet 92bcde4a41fd4096afbe97ae6e7a51ca (table test-table [id=19109218459c49d996c1d5f02345e385]) successfully deleted
I20260621 02:50:59.171386 17836 catalog_manager.cc:4841] TS 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299): tablet 92bcde4a41fd4096afbe97ae6e7a51ca (table test-table [id=19109218459c49d996c1d5f02345e385]) successfully deleted
I20260621 02:50:59.171425 17834 catalog_manager.cc:4841] TS e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383): tablet 92bcde4a41fd4096afbe97ae6e7a51ca (table test-table [id=19109218459c49d996c1d5f02345e385]) successfully deleted
W20260621 02:50:59.171868 17836 catalog_manager.cc:4808] TS 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299): delete failed for tablet 92bcde4a41fd4096afbe97ae6e7a51ca because the tablet was not found. No further retry: Not found: Tablet not found: 92bcde4a41fd4096afbe97ae6e7a51ca
I20260621 02:50:59.172065 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 17817
I20260621 02:50:59.214742 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:59.199440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.11.109.60:43443)
0621 02:50:59.199614 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:59.199618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:59.199633 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:50:59.199872 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:59.199878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:59.199983 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:59.200094 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:59.201988 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.202965 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.204451 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.205045 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.205192 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:59.207998 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:59.208006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:59.208011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:59.208047 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:59.211555 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.212400 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.212407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.212412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.212553 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.212987 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.212990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.212993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.213337 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:59.213589 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:59.214168 (+ 579us) server_negotiation.cc:299] Negotiation successful
0621 02:50:59.214379 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:50:59.215821 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:59.198004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.11.109.61:42527)
0621 02:50:59.198232 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:59.198241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:59.198333 (+ 92us) server_negotiation.cc:407] Connection header received
0621 02:50:59.198552 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:59.198561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:59.198711 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:59.198833 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:59.201559 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.202708 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.204691 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:59.205358 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:59.205465 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:59.209367 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:50:59.209375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:50:59.209380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:50:59.209433 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:50:59.212691 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.213618 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.213624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.213627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.213754 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:50:59.214175 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:50:59.214181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:50:59.214184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:50:59.214513 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:50:59.214755 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:59.215432 (+ 677us) server_negotiation.cc:299] Negotiation successful
0621 02:50:59.215608 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:50:59.991328 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:59.990371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:53561)
0621 02:50:59.990929 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0621 02:50:59.991086 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":338,"threads_started":1}
W20260621 02:50:59.991847 18963 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:36053 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:00.007802 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.007060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:46367)
0621 02:51:00.007520 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.007652 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260621 02:51:00.007938 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.007162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:54633)
0621 02:51:00.007595 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.007737 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
W20260621 02:51:00.008258 18814 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:36053 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
W20260621 02:51:00.008340 19104 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:36053 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:00.505055 19174 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b547e792b86941398e04e3f6fb122765)
I20260621 02:51:00.505404 19174 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:00.506585 19175 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b547e792b86941398e04e3f6fb122765)
I20260621 02:51:00.506763 19174 leader_election.cc:288] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053), d4aa724987c54feab6038e6ce99b67f5 (127.11.109.61:42527)
I20260621 02:51:00.506862 19175 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:00.508040 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.506866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40720)
0621 02:51:00.507536 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.507710 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":388,"threads_started":1}
I20260621 02:51:00.509517 19175 leader_election.cc:288] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053), bae160b446ca4b27b3b2693d5141436d (127.11.109.60:43443)
W20260621 02:51:00.509497 18619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:00.510692 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.508302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40734)
0621 02:51:00.510180 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.510365 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1774,"thread_start_us":1573,"threads_started":1}
W20260621 02:51:00.512041 18434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:00.521284 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.507627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:42527 (local address 127.0.0.1:44674)
0621 02:51:00.508725 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.508748 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:00.508901 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:00.512368 (+ 3467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:00.512376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:00.512415 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:00.512778 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:00.512793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.515047 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.515056 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:00.517822 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:00.517842 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.520143 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.520152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:00.520436 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:00.520490 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 02:51:00.520575 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 02:51:00.520760 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"spinlock_wait_cycles":13184,"thread_start_us":1615,"threads_started":1}
I20260621 02:51:00.521323 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.510334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:43443 (local address 127.0.0.1:38576)
0621 02:51:00.511104 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.511131 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:00.511305 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:00.512846 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:00.512853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:00.512886 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:00.513292 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:00.513309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.515316 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.515324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:00.518116 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:00.518135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.520454 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.520459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:00.520641 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:00.520675 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:51:00.520764 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 02:51:00.520928 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":1421,"threads_started":1}
I20260621 02:51:00.521781 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.520677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40750)
0621 02:51:00.521534 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.521622 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":763}
I20260621 02:51:00.522241 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.510812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.11.109.60:43443)
0621 02:51:00.512317 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:00.512322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:00.512348 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:51:00.512418 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:00.512424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:00.512503 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:00.512629 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:00.513516 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.515083 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.518419 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.520113 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.520238 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:00.520357 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:00.521735 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0621 02:51:00.521876 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1383,"thread_start_us":1588,"threads_started":1}
I20260621 02:51:00.522408 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.521985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40764)
0621 02:51:00.522130 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.522208 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:51:00.521837 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.507733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.11.109.61:42527)
0621 02:51:00.511846 (+ 4113us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:00.511855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:00.511879 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:51:00.511956 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:00.511963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:00.512089 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:00.512937 (+ 848us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:00.513292 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.514860 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.518098 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:00.519886 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:00.521170 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:00.521317 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:00.521369 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 02:51:00.521510 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3970,"thread_start_us":3779,"threads_started":1}
W20260621 02:51:00.522324 18434 leader_election.cc:334] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
W20260621 02:51:00.522650 18619 leader_election.cc:334] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:00.522846 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4aa724987c54feab6038e6ce99b67f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bae160b446ca4b27b3b2693d5141436d" is_pre_election: true
I20260621 02:51:00.523172 18643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4aa724987c54feab6038e6ce99b67f5 in term 1.
I20260621 02:51:00.523101 18457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bae160b446ca4b27b3b2693d5141436d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d4aa724987c54feab6038e6ce99b67f5" is_pre_election: true
I20260621 02:51:00.523407 18457 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae160b446ca4b27b3b2693d5141436d in term 1.
I20260621 02:51:00.523741 18434 leader_election.cc:302] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bae160b446ca4b27b3b2693d5141436d, d4aa724987c54feab6038e6ce99b67f5; no voters: b547e792b86941398e04e3f6fb122765
I20260621 02:51:00.523857 18619 leader_election.cc:302] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bae160b446ca4b27b3b2693d5141436d, d4aa724987c54feab6038e6ce99b67f5; no voters: b547e792b86941398e04e3f6fb122765
I20260621 02:51:00.524111 19175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 02:51:00.524186 19174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 02:51:00.524230 19175 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b547e792b86941398e04e3f6fb122765)
I20260621 02:51:00.524297 19174 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Starting leader election (detected failure of leader b547e792b86941398e04e3f6fb122765)
I20260621 02:51:00.524315 19175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:51:00.524387 19174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:51:00.526468 19174 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:00.526480 19175 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:00.527079 19174 leader_election.cc:288] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [CANDIDATE]: Term 2 election: Requested vote from peers b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053), d4aa724987c54feab6038e6ce99b67f5 (127.11.109.61:42527)
I20260621 02:51:00.527091 19175 leader_election.cc:288] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 2 election: Requested vote from peers b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053), bae160b446ca4b27b3b2693d5141436d (127.11.109.60:43443)
I20260621 02:51:00.527570 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.527156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40768)
0621 02:51:00.527317 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.527406 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 02:51:00.527671 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.527172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40780)
0621 02:51:00.527377 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.527474 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260621 02:51:00.527714 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4aa724987c54feab6038e6ce99b67f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bae160b446ca4b27b3b2693d5141436d"
I20260621 02:51:00.527973 18643 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4aa724987c54feab6038e6ce99b67f5 in current term 2: Already voted for candidate bae160b446ca4b27b3b2693d5141436d in this term.
I20260621 02:51:00.528095 18457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bae160b446ca4b27b3b2693d5141436d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d4aa724987c54feab6038e6ce99b67f5"
I20260621 02:51:00.528376 18457 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bae160b446ca4b27b3b2693d5141436d in current term 2: Already voted for candidate d4aa724987c54feab6038e6ce99b67f5 in this term.
I20260621 02:51:00.528810 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.528434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40794)
0621 02:51:00.528608 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.528684 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 02:51:00.528868 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.528511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:40798)
0621 02:51:00.528649 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.528718 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260621 02:51:00.529028 18434 leader_election.cc:334] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
W20260621 02:51:00.529124 18619 leader_election.cc:334] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:00.529207 18434 leader_election.cc:302] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4aa724987c54feab6038e6ce99b67f5; no voters: b547e792b86941398e04e3f6fb122765, bae160b446ca4b27b3b2693d5141436d
I20260621 02:51:00.529457 19175 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260621 02:51:00.529474 18619 leader_election.cc:302] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bae160b446ca4b27b3b2693d5141436d; no voters: b547e792b86941398e04e3f6fb122765, d4aa724987c54feab6038e6ce99b67f5
I20260621 02:51:00.529783 19174 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260621 02:51:00.993341 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:00.992510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:59315)
0621 02:51:00.992980 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:00.993081 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":250,"threads_started":1}
I20260621 02:51:01.009773 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:01.008944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:38011)
0621 02:51:01.009450 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:01.009559 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260621 02:51:01.009814 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:01.008944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:38721)
0621 02:51:01.009510 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:01.009603 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":357,"threads_started":1}
I20260621 02:51:01.995180 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:01.994323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:54131)
0621 02:51:01.994815 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:01.994983 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260621 02:51:02.011346 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.010636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:38901)
0621 02:51:02.011060 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.011176 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":236,"threads_started":1}
I20260621 02:51:02.011422 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.010572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:44593)
0621 02:51:02.011068 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.011227 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":280,"threads_started":1}
I20260621 02:51:02.181517 19190 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:02.181690 19190 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:02.182296 19190 leader_election.cc:288] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053), bae160b446ca4b27b3b2693d5141436d (127.11.109.60:43443)
I20260621 02:51:02.183185 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.182445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:41988)
0621 02:51:02.182870 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.182969 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":205,"threads_started":1}
I20260621 02:51:02.183274 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4aa724987c54feab6038e6ce99b67f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bae160b446ca4b27b3b2693d5141436d" is_pre_election: true
I20260621 02:51:02.183533 18643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4aa724987c54feab6038e6ce99b67f5 in term 2.
I20260621 02:51:02.184005 18434 leader_election.cc:302] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bae160b446ca4b27b3b2693d5141436d, d4aa724987c54feab6038e6ce99b67f5; no voters:
I20260621 02:51:02.184329 19190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260621 02:51:02.184456 19190 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:02.184593 19190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:51:02.184901 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.184506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42000)
0621 02:51:02.184705 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.184778 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:51:02.185092 18434 leader_election.cc:334] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:02.186780 19190 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:02.187285 19190 leader_election.cc:288] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 3 election: Requested vote from peers b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053), bae160b446ca4b27b3b2693d5141436d (127.11.109.60:43443)
I20260621 02:51:02.187692 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.187343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42012)
0621 02:51:02.187485 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.187546 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 02:51:02.187819 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4aa724987c54feab6038e6ce99b67f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bae160b446ca4b27b3b2693d5141436d"
I20260621 02:51:02.188016 18643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:51:02.188567 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.188253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42026)
0621 02:51:02.188381 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.188456 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 02:51:02.188746 18434 leader_election.cc:334] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
I20260621 02:51:02.190281 18643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4aa724987c54feab6038e6ce99b67f5 in term 3.
I20260621 02:51:02.190653 18434 leader_election.cc:302] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bae160b446ca4b27b3b2693d5141436d, d4aa724987c54feab6038e6ce99b67f5; no voters: b547e792b86941398e04e3f6fb122765
I20260621 02:51:02.190901 19190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:51:02.191275 19190 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [term 3 LEADER]: Becoming Leader. State: Replica: d4aa724987c54feab6038e6ce99b67f5, State: Running, Role: LEADER
I20260621 02:51:02.191637 19190 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } }
I20260621 02:51:02.193107 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4aa724987c54feab6038e6ce99b67f5. Latest consensus state: current_term: 3 leader_uuid: "d4aa724987c54feab6038e6ce99b67f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:51:02.193410 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:02.194057 19195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:02.197535 19195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:02.198308 19195 catalog_manager.cc:1259] Loaded cluster ID: 29e1e9b8c1bf476fbc48745f001454d0
I20260621 02:51:02.198401 19195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:02.199496 19195 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:02.200332 19195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Loaded TSK: 0
I20260621 02:51:02.201282 19195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:51:02.250974 18447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:51:02.257834 18447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:02.261987 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.261647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42050)
0621 02:51:02.261776 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.261849 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 02:51:02.262439 18643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [term 3 FOLLOWER]: Refusing update from remote peer d4aa724987c54feab6038e6ce99b67f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260621 02:51:02.263106 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.262721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42060)
0621 02:51:02.262891 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.262975 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 02:51:02.263180 19193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260621 02:51:02.263475 18434 consensus_peers.cc:487] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 -> Peer b547e792b86941398e04e3f6fb122765 (127.11.109.62:36053): Couldn't send request to peer b547e792b86941398e04e3f6fb122765. Status: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 02:51:02.265945 19197 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4aa724987c54feab6038e6ce99b67f5. Latest consensus state: current_term: 3 leader_uuid: "d4aa724987c54feab6038e6ce99b67f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:51:02.266175 19197 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:51:02.267309 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d4aa724987c54feab6038e6ce99b67f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:51:02.267594 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:02.268108 19197 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d4aa724987c54feab6038e6ce99b67f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b547e792b86941398e04e3f6fb122765" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36053 } } peers { permanent_uuid: "d4aa724987c54feab6038e6ce99b67f5" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "bae160b446ca4b27b3b2693d5141436d" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 43443 } attrs { promote: false } } }
I20260621 02:51:02.268363 19197 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae160b446ca4b27b3b2693d5141436d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:51:02.281895 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.275336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35659 (local address 127.0.0.1:58342)
0621 02:51:02.275921 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.275942 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:02.276061 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:02.276463 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:02.276468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:02.276490 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:02.276849 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.276858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.278449 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.278457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.279744 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.279755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.280924 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.280930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.281162 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.281226 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 02:51:02.281294 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 02:51:02.281438 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":317,"threads_started":1}
I20260621 02:51:02.282814 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.275477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.11.109.3:35659)
0621 02:51:02.276028 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.276033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.276052 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:51:02.276188 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.276195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.276305 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.276982 (+ 677us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:02.277172 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.278309 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.279887 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.280722 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.282198 (+ 1476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.282295 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.282360 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.282448 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":288,"threads_started":1}
I20260621 02:51:02.283602 19038 tablet_service.cc:1467] Processing CreateTablet for tablet b51b0e2e26c448a3b16892ff52344c1b (DEFAULT_TABLE table=test-table [id=021122801f0a42e6b8efacf2dc6ed9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:02.284003 19038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51b0e2e26c448a3b16892ff52344c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:02.296972 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.276033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35299 (local address 127.0.0.1:46432)
0621 02:51:02.276521 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.276549 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:02.276649 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:02.289727 (+ 13078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:02.289734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:02.289786 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:02.290354 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.290369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.292182 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.292187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.294441 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.294454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.296187 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.296194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.296384 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.296421 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:51:02.296489 (+ 68us) client_negotiation.cc:240] Negotiation successful
0621 02:51:02.296644 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":684,"threads_started":1}
I20260621 02:51:02.297986 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.276436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.11.109.1:35299)
0621 02:51:02.289065 (+ 12629us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.289074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.289097 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:51:02.289211 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.289219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.289329 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.290554 (+ 1225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:02.290936 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.292006 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.294734 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.295962 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.297327 (+ 1365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.297431 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.297482 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.297625 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12455,"thread_start_us":13018,"threads_started":1}
I20260621 02:51:02.298641 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.274875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36383 (local address 127.0.0.1:60780)
0621 02:51:02.275089 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.275114 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:02.275219 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:02.291835 (+ 16616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:02.291840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:02.291864 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:02.292202 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.292216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.293813 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.293818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.296176 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.296188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.297824 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.297831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.297996 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.298035 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:51:02.298179 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 02:51:02.298304 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260621 02:51:02.298728 18747 tablet_service.cc:1467] Processing CreateTablet for tablet b51b0e2e26c448a3b16892ff52344c1b (DEFAULT_TABLE table=test-table [id=021122801f0a42e6b8efacf2dc6ed9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:02.299247 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51b0e2e26c448a3b16892ff52344c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:02.299641 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.274993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.11.109.2:36383)
0621 02:51:02.291429 (+ 16436us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.291435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.291453 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:51:02.291519 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.291524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.291595 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.292352 (+ 757us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:02.292553 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.293669 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.296368 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.297594 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.298945 (+ 1351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.299039 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.299226 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.299370 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16317,"thread_start_us":16692,"threads_started":1}
I20260621 02:51:02.300313 18897 tablet_service.cc:1467] Processing CreateTablet for tablet b51b0e2e26c448a3b16892ff52344c1b (DEFAULT_TABLE table=test-table [id=021122801f0a42e6b8efacf2dc6ed9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:02.300714 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51b0e2e26c448a3b16892ff52344c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:02.307606 19208 tablet_bootstrap.cc:492] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Bootstrap starting.
I20260621 02:51:02.309703 19210 tablet_bootstrap.cc:492] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Bootstrap starting.
I20260621 02:51:02.310175 19208 tablet_bootstrap.cc:654] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:02.311470 19210 tablet_bootstrap.cc:654] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:02.314553 19210 tablet_bootstrap.cc:492] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: No bootstrap required, opened a new log
I20260621 02:51:02.314747 19210 ts_tablet_manager.cc:1397] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:51:02.315763 19210 raft_consensus.cc:348] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.316068 19210 raft_consensus.cc:374] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:02.316305 19210 raft_consensus.cc:729] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76cd48a4a9a48f6ac96f2bee324c7ea, State: Initialized, Role: FOLLOWER
I20260621 02:51:02.316728 19210 consensus_queue.cc:260] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.317936 19209 tablet_bootstrap.cc:492] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Bootstrap starting.
I20260621 02:51:02.318382 19210 ts_tablet_manager.cc:1428] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260621 02:51:02.318521 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.318088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:36327)
0621 02:51:02.318241 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.318349 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260621 02:51:02.319466 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.319071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:49503)
0621 02:51:02.319237 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.319320 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 02:51:02.320317 19208 tablet_bootstrap.cc:492] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: No bootstrap required, opened a new log
I20260621 02:51:02.320465 19208 ts_tablet_manager.cc:1397] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260621 02:51:02.321311 19208 raft_consensus.cc:348] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.321549 19208 raft_consensus.cc:374] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:02.321645 19208 raft_consensus.cc:729] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a15586458ed42b2ae80e2dbc4b6d669, State: Initialized, Role: FOLLOWER
I20260621 02:51:02.321919 19208 consensus_queue.cc:260] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.322746 19208 ts_tablet_manager.cc:1428] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 02:51:02.322790 19209 tablet_bootstrap.cc:654] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:02.323392 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.323047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:41385)
0621 02:51:02.323191 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.323282 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 02:51:02.324321 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.323928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:42787)
0621 02:51:02.324080 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.324177 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:51:02.325734 19209 tablet_bootstrap.cc:492] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: No bootstrap required, opened a new log
I20260621 02:51:02.325729 19105 heartbeater.cc:503] Master 127.11.109.61:42527 was elected leader, sending a full tablet report...
I20260621 02:51:02.325889 19209 ts_tablet_manager.cc:1397] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260621 02:51:02.326731 19209 raft_consensus.cc:348] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.326926 19209 raft_consensus.cc:374] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:02.327025 19209 raft_consensus.cc:729] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ba662c9ec34515bc826593508a9c48, State: Initialized, Role: FOLLOWER
I20260621 02:51:02.327296 19209 consensus_queue.cc:260] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.328202 19209 ts_tablet_manager.cc:1428] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 02:51:02.329329 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.328903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:54113)
0621 02:51:02.329047 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.329138 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 02:51:02.329866 18815 heartbeater.cc:503] Master 127.11.109.61:42527 was elected leader, sending a full tablet report...
I20260621 02:51:02.330475 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.329915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:41267)
0621 02:51:02.330189 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.330275 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260621 02:51:02.415429 19214 raft_consensus.cc:482] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:02.415712 19214 raft_consensus.cc:504] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.417383 19214 leader_election.cc:288] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383), 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:51:02.423607 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.417636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:50521 (local address 127.11.109.2:36383)
0621 02:51:02.417884 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.417892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.417920 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:51:02.418151 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.418157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.418228 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.418334 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:02.418859 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.419723 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.421979 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.422753 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.422923 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.422993 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.423238 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.423359 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:51:02.423661 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.417522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36383 (local address 127.11.109.3:50521)
0621 02:51:02.417817 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.417837 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:02.417965 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:02.418354 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:02.418358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:02.418384 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:02.418692 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.418704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.419869 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.419875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.421817 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.421834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.422923 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.422929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.423086 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.423108 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:51:02.423213 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:51:02.423353 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":76}
I20260621 02:51:02.424566 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.417837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:49941 (local address 127.11.109.1:35299)
0621 02:51:02.418020 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.418027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.418536 (+ 509us) server_negotiation.cc:407] Connection header received
0621 02:51:02.418675 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.418680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.418759 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.418849 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:02.419366 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.420243 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.422075 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.423278 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.423424 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.423489 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.424163 (+ 674us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.424308 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:51:02.424425 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51b0e2e26c448a3b16892ff52344c1b" candidate_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" is_pre_election: true
I20260621 02:51:02.424625 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.417631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35299 (local address 127.11.109.3:49941)
0621 02:51:02.418433 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.418447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:02.418554 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:02.418873 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:02.418877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:02.418900 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:02.419214 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.419225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.420372 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.420375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.421878 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:02.421909 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.423631 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.423638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:02.423978 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.424026 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 02:51:02.424129 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:51:02.424248 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":19,"thread_start_us":459,"threads_started":1}
I20260621 02:51:02.424842 18917 raft_consensus.cc:2454] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a15586458ed42b2ae80e2dbc4b6d669 in term 0.
I20260621 02:51:02.425226 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51b0e2e26c448a3b16892ff52344c1b" candidate_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ba662c9ec34515bc826593508a9c48" is_pre_election: true
I20260621 02:51:02.425402 18992 leader_election.cc:302] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a15586458ed42b2ae80e2dbc4b6d669, e76cd48a4a9a48f6ac96f2bee324c7ea; no voters:
I20260621 02:51:02.425499 18768 raft_consensus.cc:2454] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a15586458ed42b2ae80e2dbc4b6d669 in term 0.
I20260621 02:51:02.425810 19214 raft_consensus.cc:2781] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:02.425940 19214 raft_consensus.cc:482] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:02.426048 19214 raft_consensus.cc:3037] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:02.428028 19214 raft_consensus.cc:504] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.428798 19214 leader_election.cc:288] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [CANDIDATE]: Term 1 election: Requested vote from peers e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383), 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299)
I20260621 02:51:02.428997 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51b0e2e26c448a3b16892ff52344c1b" candidate_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea"
I20260621 02:51:02.429111 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51b0e2e26c448a3b16892ff52344c1b" candidate_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58ba662c9ec34515bc826593508a9c48"
I20260621 02:51:02.429214 18917 raft_consensus.cc:3037] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:02.429323 18768 raft_consensus.cc:3037] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:02.431938 18768 raft_consensus.cc:2454] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a15586458ed42b2ae80e2dbc4b6d669 in term 1.
I20260621 02:51:02.432196 18917 raft_consensus.cc:2454] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a15586458ed42b2ae80e2dbc4b6d669 in term 1.
I20260621 02:51:02.432366 18994 leader_election.cc:302] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a15586458ed42b2ae80e2dbc4b6d669, 58ba662c9ec34515bc826593508a9c48; no voters:
I20260621 02:51:02.432713 19214 raft_consensus.cc:2781] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:02.433149 19214 raft_consensus.cc:686] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 LEADER]: Becoming Leader. State: Replica: 2a15586458ed42b2ae80e2dbc4b6d669, State: Running, Role: LEADER
I20260621 02:51:02.433598 19214 consensus_queue.cc:237] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } }
I20260621 02:51:02.434398 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.433973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:34709)
0621 02:51:02.434131 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.434238 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 02:51:02.437265 18447 catalog_manager.cc:5495] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a15586458ed42b2ae80e2dbc4b6d669" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 } health_report { overall_health: UNKNOWN } } }
I20260621 02:51:02.478596 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.473922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.11.109.3:35659)
0621 02:51:02.474101 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.474106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.474122 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:51:02.474307 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.474311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.474401 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.474517 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:02.474802 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.475512 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.476706 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.477096 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.477245 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.477557 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:02.478065 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:02.478186 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.478245 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.478302 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:51:02.482496 18768 raft_consensus.cc:1261] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Refusing update from remote peer 2a15586458ed42b2ae80e2dbc4b6d669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:02.482837 18917 raft_consensus.cc:1261] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Refusing update from remote peer 2a15586458ed42b2ae80e2dbc4b6d669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:02.483213 19218 consensus_queue.cc:1035] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58ba662c9ec34515bc826593508a9c48" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:02.483598 19214 consensus_queue.cc:1035] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76cd48a4a9a48f6ac96f2bee324c7ea" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:02.485790 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.485354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:57715)
0621 02:51:02.485521 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.485618 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:51:02.486245 18964 heartbeater.cc:503] Master 127.11.109.61:42527 was elected leader, sending a full tablet report...
I20260621 02:51:02.486922 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.486476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:37921)
0621 02:51:02.486685 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.486782 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260621 02:51:02.492108 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.491408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.1:33849)
0621 02:51:02.491778 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.491883 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260621 02:51:02.495427 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.495041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:58135)
0621 02:51:02.495183 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.495276 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260621 02:51:02.502786 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.502363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.3:44997)
0621 02:51:02.502543 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.502644 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 02:51:02.503822 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.503168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.11.109.2:55271)
0621 02:51:02.503466 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.503575 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260621 02:51:02.503948 19222 mvcc.cc:205] Tried to move back new op lower bound from 7299114035121119232 to 7299114034932436992. Current Snapshot: MvccSnapshot[applied={T|T < 7299114035121119232}]
I20260621 02:51:02.509403 18447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:02.509708 18447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:02.515780 18447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4aa724987c54feab6038e6ce99b67f5: Sending DeleteTablet for 3 replicas of tablet b51b0e2e26c448a3b16892ff52344c1b
I20260621 02:51:02.516882 18897 tablet_service.cc:1514] Processing DeleteTablet for tablet b51b0e2e26c448a3b16892ff52344c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:02 UTC) from {username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:60780
I20260621 02:51:02.516898 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet b51b0e2e26c448a3b16892ff52344c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:02 UTC) from {username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:46432
I20260621 02:51:02.517040 18896 tablet_service.cc:1514] Processing DeleteTablet for tablet b51b0e2e26c448a3b16892ff52344c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:02 UTC) from {username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:60780
I20260621 02:51:02.517356 19038 tablet_service.cc:1514] Processing DeleteTablet for tablet b51b0e2e26c448a3b16892ff52344c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:02 UTC) from {username='oryx', principal='oryx/127.11.109.61@KRBTEST.COM'} at 127.0.0.1:58342
I20260621 02:51:02.517714 19230 tablet_replica.cc:330] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: stopping tablet replica
I20260621 02:51:02.517951 19230 raft_consensus.cc:2229] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:02.518383 19230 raft_consensus.cc:2258] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:02.519773 19230 ts_tablet_manager.cc:1905] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:02.519924 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18620
I20260621 02:51:02.526593 19230 ts_tablet_manager.cc:1918] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:02.526826 19230 log.cc:1196] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/b51b0e2e26c448a3b16892ff52344c1b
I20260621 02:51:02.527341 19230 ts_tablet_manager.cc:1939] T b51b0e2e26c448a3b16892ff52344c1b P 58ba662c9ec34515bc826593508a9c48: Deleting consensus metadata
I20260621 02:51:02.528512 18434 catalog_manager.cc:4841] TS 58ba662c9ec34515bc826593508a9c48 (127.11.109.1:35299): tablet b51b0e2e26c448a3b16892ff52344c1b (table test-table [id=021122801f0a42e6b8efacf2dc6ed9ba]) successfully deleted
I20260621 02:51:02.530264 19231 tablet_replica.cc:330] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: stopping tablet replica
I20260621 02:51:02.530517 19231 raft_consensus.cc:2229] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:02.530846 19231 raft_consensus.cc:2258] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:02.534021 19232 tablet_replica.cc:330] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: stopping tablet replica
I20260621 02:51:02.534045 19231 ts_tablet_manager.cc:1905] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:02.534209 19232 raft_consensus.cc:2229] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:51:02.534804 19232 raft_consensus.cc:2258] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:02.535784 19232 ts_tablet_manager.cc:1905] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:02.540923 19231 ts_tablet_manager.cc:1918] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:02.541133 19231 log.cc:1196] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/b51b0e2e26c448a3b16892ff52344c1b
I20260621 02:51:02.541677 19231 ts_tablet_manager.cc:1939] T b51b0e2e26c448a3b16892ff52344c1b P e76cd48a4a9a48f6ac96f2bee324c7ea: Deleting consensus metadata
I20260621 02:51:02.542794 18432 catalog_manager.cc:4841] TS e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383): tablet b51b0e2e26c448a3b16892ff52344c1b (table test-table [id=021122801f0a42e6b8efacf2dc6ed9ba]) successfully deleted
I20260621 02:51:02.542889 19232 ts_tablet_manager.cc:1918] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:02.543008 19232 log.cc:1196] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/b51b0e2e26c448a3b16892ff52344c1b
I20260621 02:51:02.543555 19232 ts_tablet_manager.cc:1939] T b51b0e2e26c448a3b16892ff52344c1b P 2a15586458ed42b2ae80e2dbc4b6d669: Deleting consensus metadata
W20260621 02:51:02.543788 18432 catalog_manager.cc:4808] TS e76cd48a4a9a48f6ac96f2bee324c7ea (127.11.109.2:36383): delete failed for tablet b51b0e2e26c448a3b16892ff52344c1b because the tablet was not found. No further retry: Not found: Tablet not found: b51b0e2e26c448a3b16892ff52344c1b
I20260621 02:51:02.544582 18434 catalog_manager.cc:4841] TS 2a15586458ed42b2ae80e2dbc4b6d669 (127.11.109.3:35659): tablet b51b0e2e26c448a3b16892ff52344c1b (table test-table [id=021122801f0a42e6b8efacf2dc6ed9ba]) successfully deleted
I20260621 02:51:02.548031 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.521349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.11.109.1:35299)
0621 02:51:02.521514 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.521523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.529693 (+ 8170us) server_negotiation.cc:407] Connection header received
0621 02:51:02.529890 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.529897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.529992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.530114 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:02.533028 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.534315 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.536622 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.537283 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.537371 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.541414 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:02.541431 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:02.541452 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:02.541494 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:02.545099 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:02.545903 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:02.545910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:02.545912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:02.546024 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:02.546389 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:02.546392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:02.546394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:02.546814 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:02.547129 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.547494 (+ 365us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.547708 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 02:51:02.548789 18808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:02.653955 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18620
I20260621 02:51:02.671319 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18825
I20260621 02:51:02.684701 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.672141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.11.109.2:36383)
0621 02:51:02.672344 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.672350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.672370 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:51:02.672509 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.672517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.672597 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.672691 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:02.674067 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.674829 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.676091 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.676495 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.676625 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.679130 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:02.679143 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:02.679159 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:02.679188 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:02.682114 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:02.682771 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:02.682784 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:02.682786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:02.682905 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:02.683273 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:02.683278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:02.683282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:02.683612 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:02.683918 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.684173 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.684392 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:51:02.685482 18957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:02.695479 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.694929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42064)
0621 02:51:02.695144 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.695291 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 02:51:02.696609 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.696178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36053 (local address 127.0.0.1:42072)
0621 02:51:02.696337 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:02.696452 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:51:02.803951 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18825
I20260621 02:51:02.827592 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18975
I20260621 02:51:02.843326 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:02.828428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.11.109.3:35659)
0621 02:51:02.828595 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:02.828599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:02.828731 (+ 132us) server_negotiation.cc:407] Connection header received
0621 02:51:02.828929 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:02.828933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:02.828997 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:02.829085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:02.830978 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.831761 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.833554 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:02.833980 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:02.834072 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:02.837421 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:02.837437 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:02.837453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:02.837484 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:02.840556 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:02.841359 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:02.841366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:02.841369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:02.841503 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:02.841916 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:02.841919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:02.841920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:02.842257 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:02.842474 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:02.842838 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 02:51:02.843001 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:51:02.844059 19098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:02.970927 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18975
I20260621 02:51:02.993000 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18310
I20260621 02:51:02.994031 18477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:03.112592 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18310
W20260621 02:51:03.133285 18617 connection.cc:535] server connection from 127.0.0.1:38576 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 02:51:03.133602 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 18411
I20260621 02:51:03.134691 18663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:03.246021 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 18411
2026-06-21T02:51:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10982 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:51:03 dist-test-slave-bc52 krb5kdc[19241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:51:03 dist-test-slave-bc52 krb5kdc[19241](info): set up 2 sockets
Jun 21 02:51:03 dist-test-slave-bc52 krb5kdc[19241](info): commencing operation
krb5kdc: starting...
W20260621 02:51:04.354130 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:51:04 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab.
Entry for principal oryx/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab.
I20260621 02:51:04.556056 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:36623
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab
--principal=oryx/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:36623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:04.864944 19254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:04.865293 19254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:04.865449 19254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:04.872402 19254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:04.872556 19254 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:51:04.872671 19254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:04.872797 19254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:04.872881 19254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:04.872968 19254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:04.882440 19254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:36623
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.62.keytab
--principal=oryx/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:36623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:04.883670 19254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:04.884966 19254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:04.896238 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:04.896270 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:04.896173 19262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:04.897344 19254 server_base.cc:1034] running on GCE node
I20260621 02:51:04.898038 19254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:04.899356 19254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:04.900568 19254 hybrid_clock.cc:648] HybridClock initialized: now 1782010264900513 us; error 39 us; skew 500 ppm
I20260621 02:51:04.900885 19254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:04 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010264, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:04.903918 19254 init.cc:377] Logged in from keytab as oryx/127.11.109.62@KRBTEST.COM (short username oryx)
I20260621 02:51:04.905323 19254 webserver.cc:460] Webserver started at http://127.11.109.62:37589/ using document root <none> and password file <none>
I20260621 02:51:04.905973 19254 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:04.906080 19254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:04.906416 19254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:04.909503 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "9be0b377e87746fca9c7ac4bdf325a5e"
format_stamp: "Formatted at 2026-06-21 02:51:04 on dist-test-slave-bc52"
server_key: "2c1c146678f65d53c8b9e4d99bbfc589"
server_key_iv: "d71fc704e31a28a7298740972c511f11"
server_key_version: "encryptionkey@0"
I20260621 02:51:04.910450 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "9be0b377e87746fca9c7ac4bdf325a5e"
format_stamp: "Formatted at 2026-06-21 02:51:04 on dist-test-slave-bc52"
server_key: "2c1c146678f65d53c8b9e4d99bbfc589"
server_key_iv: "d71fc704e31a28a7298740972c511f11"
server_key_version: "encryptionkey@0"
I20260621 02:51:04.915163 19254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:51:04.918323 19269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:04.919552 19254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:51:04.919677 19254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "9be0b377e87746fca9c7ac4bdf325a5e"
format_stamp: "Formatted at 2026-06-21 02:51:04 on dist-test-slave-bc52"
server_key: "2c1c146678f65d53c8b9e4d99bbfc589"
server_key_iv: "d71fc704e31a28a7298740972c511f11"
server_key_version: "encryptionkey@0"
I20260621 02:51:04.919845 19254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:04.936340 19254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:04.940066 19254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:04.940296 19254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:04.960806 19254 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:36623
I20260621 02:51:04.960813 19330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:36623 every 8 connection(s)
I20260621 02:51:04.962239 19254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:51:04.963976 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 19254
I20260621 02:51:04.964186 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:51:04.964946 11700 external_mini_cluster.cc:1428] Setting key 06363e4c52dc7779e293cef3b195efa3
I20260621 02:51:04.966158 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:51:04 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:04.979925 19331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Bootstrap starting.
I20260621 02:51:04.982999 19331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:04.983970 19331 log.cc:824] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:04.986759 19331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: No bootstrap required, opened a new log
I20260621 02:51:04.986915 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:04.966898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.11.109.62:36623)
0621 02:51:04.967646 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:04.967657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:04.967695 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:51:04.969162 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:04.969231 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:04.969607 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:04.969915 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:04.971650 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:04.972937 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:04.974721 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:04.975338 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:04.975462 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:04.979981 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:04.979988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:04.980023 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:04.980074 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:04.983093 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:04.983786 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:04.983799 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:04.983805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:04.983964 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:04.984339 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:04.984345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:04.984348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:04.984851 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:04.985165 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:04.985608 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 02:51:04.985997 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260621 02:51:04.994379 19331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36623 } }
I20260621 02:51:04.994799 19331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:04.994951 19331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be0b377e87746fca9c7ac4bdf325a5e, State: Initialized, Role: FOLLOWER
I20260621 02:51:04.995530 19331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36623 } }
I20260621 02:51:04.995754 19331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:51:04.995877 19331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:51:04.996053 19331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:04.998149 19331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36623 } }
I20260621 02:51:04.998661 19331 leader_election.cc:302] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9be0b377e87746fca9c7ac4bdf325a5e; no voters:
I20260621 02:51:04.999228 19331 leader_election.cc:288] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:51:04.999317 19336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:05.000089 19336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [term 1 LEADER]: Becoming Leader. State: Replica: 9be0b377e87746fca9c7ac4bdf325a5e, State: Running, Role: LEADER
I20260621 02:51:05.000623 19336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36623 } }
I20260621 02:51:05.001235 19331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:51:05.004380 19338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be0b377e87746fca9c7ac4bdf325a5e. Latest consensus state: current_term: 1 leader_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36623 } } }
I20260621 02:51:05.004443 19337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9be0b377e87746fca9c7ac4bdf325a5e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 36623 } } }
I20260621 02:51:05.004927 19337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:05.005043 19338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e [sys.catalog]: This master's current role is: LEADER
W20260621 02:51:05.011878 19352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:51:05.012017 19352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:51:05.012152 19346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:05.013829 19346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:05.021552 19346 catalog_manager.cc:1347] Generated new cluster ID: 1273ee1801b34bc989ead9eaceeac844
I20260621 02:51:05.021656 19346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:05.029006 19346 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:51:05.030732 19346 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:05.043753 19346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Generated new TSK 0
I20260621 02:51:05.044749 19346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab.
Entry for principal oryx/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab.
I20260621 02:51:05.098641 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36623
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:05.418416 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:05.418659 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:05.418752 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:05.425869 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:05.425962 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:05.426038 19357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:51:05.426111 19357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:51:05.435179 19357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:05.436339 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:05.437654 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:05.450148 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:05.450205 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:05.450225 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:05.450965 19357 server_base.cc:1034] running on GCE node
I20260621 02:51:05.451470 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:05.452298 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:05.453532 19357 hybrid_clock.cc:648] HybridClock initialized: now 1782010265453460 us; error 53 us; skew 500 ppm
I20260621 02:51:05.453857 19357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:05 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010265, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:05.456836 19357 init.cc:377] Logged in from keytab as oryx/127.11.109.1@KRBTEST.COM (short username oryx)
I20260621 02:51:05.458149 19357 webserver.cc:460] Webserver started at http://127.11.109.1:46339/ using document root <none> and password file <none>
I20260621 02:51:05.458802 19357 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:05.458932 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:05.459254 19357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:05.462090 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
format_stamp: "Formatted at 2026-06-21 02:51:05 on dist-test-slave-bc52"
server_key: "d91397aa5865428f8a83ae2b96f8d52d"
server_key_iv: "59088b6c9e6a73e4dd8b9f6c1adee4b9"
server_key_version: "encryptionkey@0"
I20260621 02:51:05.462859 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
format_stamp: "Formatted at 2026-06-21 02:51:05 on dist-test-slave-bc52"
server_key: "d91397aa5865428f8a83ae2b96f8d52d"
server_key_iv: "59088b6c9e6a73e4dd8b9f6c1adee4b9"
server_key_version: "encryptionkey@0"
I20260621 02:51:05.467710 19357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 02:51:05.470968 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:05.472311 19357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:51:05.472476 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
format_stamp: "Formatted at 2026-06-21 02:51:05 on dist-test-slave-bc52"
server_key: "d91397aa5865428f8a83ae2b96f8d52d"
server_key_iv: "59088b6c9e6a73e4dd8b9f6c1adee4b9"
server_key_version: "encryptionkey@0"
I20260621 02:51:05.472728 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:05.494119 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:05.498126 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:05.498410 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:05.500937 19357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:05.501053 19357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:05.501276 19357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:05.501348 19357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:51:05 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010265, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:05.526865 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:05.501759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:38507 (local address 127.11.109.62:36623)
0621 02:51:05.502262 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:05.502270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:05.504050 (+ 1780us) server_negotiation.cc:407] Connection header received
0621 02:51:05.506460 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:05.506467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:05.506562 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:05.506725 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:05.509850 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.510781 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.513542 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.513940 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.514071 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:05.519051 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:05.519060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:05.519069 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:05.519120 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:05.523105 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:05.523912 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:05.523921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:05.523927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:05.524109 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:05.524616 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:05.524624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:05.524630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:05.525092 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:05.525323 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:05.525997 (+ 674us) server_negotiation.cc:299] Negotiation successful
0621 02:51:05.526305 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
I20260621 02:51:05.527380 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:05.502995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.11.109.1:38507)
0621 02:51:05.503896 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:05.503935 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:05.506100 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:05.507278 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:05.507292 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:05.508445 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:05.509483 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:05.509515 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.511058 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.511063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:05.513366 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:05.513380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.514178 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.514185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:05.514476 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:05.515335 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:05.515382 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:05.518723 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:05.523275 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:05.523305 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:05.523324 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:05.523765 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:05.524266 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:05.524271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:05.524276 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:05.524495 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:05.525266 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:05.525272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:05.525614 (+ 342us) client_negotiation.cc:769] Sending connection context
0621 02:51:05.525842 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 02:51:05.526213 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":405,"threads_started":1}
I20260621 02:51:05.555449 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:05.539392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:34687 (local address 127.11.109.62:36623)
0621 02:51:05.539612 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:05.539621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:05.540253 (+ 632us) server_negotiation.cc:407] Connection header received
0621 02:51:05.540454 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:05.540458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:05.540517 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:05.540595 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:05.542161 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.542947 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.545571 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.545975 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.546096 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:05.548753 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:05.548763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:05.548768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:05.548818 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:05.552043 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:05.553129 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:05.553136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:05.553141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:05.553326 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:05.553770 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:05.553776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:05.553780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:05.554181 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:05.554354 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:05.554871 (+ 517us) server_negotiation.cc:299] Negotiation successful
0621 02:51:05.555087 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:51:05.555572 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:05.539393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.11.109.1:34687)
0621 02:51:05.540119 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:05.540144 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:05.540327 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:05.540668 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:05.540676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:05.541448 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:05.541970 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:05.541983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.543135 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.543143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:05.545416 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:05.545437 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:05.546209 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:05.546219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:05.546467 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:05.547221 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:05.547242 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:05.548551 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:05.552263 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:05.552270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:05.552275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:05.552981 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:05.553440 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:05.553445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:05.553449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:05.553624 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:05.554356 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:05.554367 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:05.554608 (+ 241us) client_negotiation.cc:769] Sending connection context
0621 02:51:05.554769 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 02:51:05.555110 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":355,"threads_started":1}
I20260621 02:51:05.559849 19357 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:37613
I20260621 02:51:05.559854 19492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:37613 every 8 connection(s)
I20260621 02:51:05.561036 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:51:05.562999 19493 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36623
I20260621 02:51:05.563216 19493 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:05.563884 19493 heartbeater.cc:511] Master 127.11.109.62:36623 requested a full tablet report, sending...
I20260621 02:51:05.565785 19286 ts_manager.cc:194] Registered new tserver with Master: 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613)
I20260621 02:51:05.567144 19286 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:38507
I20260621 02:51:05.569005 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 19357
I20260621 02:51:05.569237 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:51:05.569981 11700 external_mini_cluster.cc:1428] Setting key f339bd80724f68a5a0a98401bcd2ff07
WARNING: no policy specified for oryx/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab.
Entry for principal oryx/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab.
I20260621 02:51:05.614012 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36623
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:05.928752 19497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:05.929033 19497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:05.929258 19497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:05.936376 19497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:05.936514 19497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:05.936645 19497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:51:05.936779 19497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:51:05.946089 19497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:05.947288 19497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:05.948598 19497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:05.961567 19503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:05.962293 19505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:05.962641 19497 server_base.cc:1034] running on GCE node
W20260621 02:51:05.965667 19502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:05.966441 19497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:05.967459 19497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:05.968713 19497 hybrid_clock.cc:648] HybridClock initialized: now 1782010265968647 us; error 35 us; skew 500 ppm
I20260621 02:51:05.969026 19497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:05 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010265, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:05.972605 19497 init.cc:377] Logged in from keytab as oryx/127.11.109.2@KRBTEST.COM (short username oryx)
I20260621 02:51:05.974046 19497 webserver.cc:460] Webserver started at http://127.11.109.2:40605/ using document root <none> and password file <none>
I20260621 02:51:05.974704 19497 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:05.974838 19497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:05.975147 19497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:05.978173 19497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "37ab78a2e85943b9820cdbf29ec3f4b3"
format_stamp: "Formatted at 2026-06-21 02:51:05 on dist-test-slave-bc52"
server_key: "102b9e8acdac6807501cf794c6ed4f5f"
server_key_iv: "e2f3de498f66a3fb0f05038d0dbfc3e4"
server_key_version: "encryptionkey@0"
I20260621 02:51:05.978951 19497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "37ab78a2e85943b9820cdbf29ec3f4b3"
format_stamp: "Formatted at 2026-06-21 02:51:05 on dist-test-slave-bc52"
server_key: "102b9e8acdac6807501cf794c6ed4f5f"
server_key_iv: "e2f3de498f66a3fb0f05038d0dbfc3e4"
server_key_version: "encryptionkey@0"
I20260621 02:51:05.983790 19497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:51:05.987037 19512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:05.988195 19497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:51:05.988361 19497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "37ab78a2e85943b9820cdbf29ec3f4b3"
format_stamp: "Formatted at 2026-06-21 02:51:05 on dist-test-slave-bc52"
server_key: "102b9e8acdac6807501cf794c6ed4f5f"
server_key_iv: "e2f3de498f66a3fb0f05038d0dbfc3e4"
server_key_version: "encryptionkey@0"
I20260621 02:51:05.988561 19497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:06.014331 19497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:06.018184 19497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:06.018456 19497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:06.020910 19497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:06.021028 19497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:06.021210 19497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:06.021284 19497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 21 02:51:06 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010265, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:06.044243 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.021718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:40463 (local address 127.11.109.62:36623)
0621 02:51:06.021914 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:06.021918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:06.023723 (+ 1805us) server_negotiation.cc:407] Connection header received
0621 02:51:06.026184 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:06.026189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:06.026277 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:06.026388 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:06.028490 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.029747 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.032248 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.033012 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.033168 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.037285 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:06.037294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:06.037310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:06.037357 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:06.040896 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.041886 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.041892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.041898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.042027 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.042471 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.042475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.042479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.042826 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:06.042978 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:06.043618 (+ 640us) server_negotiation.cc:299] Negotiation successful
0621 02:51:06.043826 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:51:06.045532 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.022631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.11.109.2:40463)
0621 02:51:06.023561 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:06.023604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:06.025866 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:06.026680 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:06.026694 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:06.027501 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:06.028328 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.028349 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.029878 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.029890 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.032052 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.032070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.033281 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.033290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.033565 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.034343 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:06.034382 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:06.037059 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:06.041055 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.041061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.041071 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.041759 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.042140 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.042145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.042149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.042353 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.042951 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:06.042958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:06.043268 (+ 310us) client_negotiation.cc:769] Sending connection context
0621 02:51:06.043512 (+ 244us) client_negotiation.cc:240] Negotiation successful
0621 02:51:06.043986 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":425,"threads_started":1}
I20260621 02:51:06.067488 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.051158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.11.109.2:59585)
0621 02:51:06.051747 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:06.051765 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:06.051913 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:06.052226 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:06.052231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:06.052667 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:06.053030 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.053040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.054665 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.054679 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.056921 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.056948 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.058187 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.058194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.058329 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.059029 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:06.059060 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:06.060406 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:06.064204 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.064212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.064219 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.064870 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.065281 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.065286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.065290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.065468 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.066018 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:06.066023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:06.066362 (+ 339us) client_negotiation.cc:769] Sending connection context
0621 02:51:06.066554 (+ 192us) client_negotiation.cc:240] Negotiation successful
0621 02:51:06.066882 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":349,"threads_started":1}
I20260621 02:51:06.068002 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.051066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:59585 (local address 127.11.109.62:36623)
0621 02:51:06.051211 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:06.051214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:06.051852 (+ 638us) server_negotiation.cc:407] Connection header received
0621 02:51:06.052043 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:06.052047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:06.052101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:06.052199 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:06.053174 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.054446 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.057216 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.057871 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.057996 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.060667 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:06.060681 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:06.060685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:06.060726 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:06.064030 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.064987 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.064991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.064993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.065107 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.065586 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.065589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.065591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.065887 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:06.066036 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:06.067496 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0621 02:51:06.067712 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:51:06.078649 19497 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:39457
I20260621 02:51:06.078685 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:39457 every 8 connection(s)
I20260621 02:51:06.079777 19497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:51:06.081664 19632 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36623
I20260621 02:51:06.081880 19632 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:06.082463 19632 heartbeater.cc:511] Master 127.11.109.62:36623 requested a full tablet report, sending...
I20260621 02:51:06.083570 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 19497
I20260621 02:51:06.083743 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:51:06.083748 19286 ts_manager.cc:194] Registered new tserver with Master: 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457)
I20260621 02:51:06.084288 11700 external_mini_cluster.cc:1428] Setting key 3a01b4a0e786422d7a36ddbeecc76575
I20260621 02:51:06.084825 19286 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:40463
WARNING: no policy specified for oryx/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab.
Entry for principal oryx/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab.
I20260621 02:51:06.128208 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:36623
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:06.441610 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:06.441897 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:06.442070 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:06.449740 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:06.449880 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:06.450026 19636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:51:06.450158 19636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:51:06.459764 19636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.109.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:06.460975 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:06.462363 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:06.475346 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:06.475445 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:06.475442 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:06.476572 19636 server_base.cc:1034] running on GCE node
I20260621 02:51:06.477273 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:06.478147 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:06.479393 19636 hybrid_clock.cc:648] HybridClock initialized: now 1782010266479328 us; error 43 us; skew 500 ppm
I20260621 02:51:06.479691 19636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:06 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010266, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:06.482689 19636 init.cc:377] Logged in from keytab as oryx/127.11.109.3@KRBTEST.COM (short username oryx)
I20260621 02:51:06.484069 19636 webserver.cc:460] Webserver started at http://127.11.109.3:42567/ using document root <none> and password file <none>
I20260621 02:51:06.484763 19636 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:06.484889 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:06.485275 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:06.488106 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
format_stamp: "Formatted at 2026-06-21 02:51:06 on dist-test-slave-bc52"
server_key: "55639f5bc02e94f73b6b091127790bb5"
server_key_iv: "677f386ff1e5b7dac781c0ac24f03100"
server_key_version: "encryptionkey@0"
I20260621 02:51:06.488888 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
format_stamp: "Formatted at 2026-06-21 02:51:06 on dist-test-slave-bc52"
server_key: "55639f5bc02e94f73b6b091127790bb5"
server_key_iv: "677f386ff1e5b7dac781c0ac24f03100"
server_key_version: "encryptionkey@0"
I20260621 02:51:06.494127 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260621 02:51:06.497386 19651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:06.498565 19636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:51:06.498749 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
format_stamp: "Formatted at 2026-06-21 02:51:06 on dist-test-slave-bc52"
server_key: "55639f5bc02e94f73b6b091127790bb5"
server_key_iv: "677f386ff1e5b7dac781c0ac24f03100"
server_key_version: "encryptionkey@0"
I20260621 02:51:06.498960 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:06.511054 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:06.514662 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:06.514930 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:06.517455 19636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:06.517563 19636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:06.517737 19636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:06.517817 19636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:51:06 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010266, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:06.542111 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.518271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:59081 (local address 127.11.109.62:36623)
0621 02:51:06.518468 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:06.518474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:06.519880 (+ 1406us) server_negotiation.cc:407] Connection header received
0621 02:51:06.522205 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:06.522214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:06.522313 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:06.522396 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:06.525142 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.526514 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.528695 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.529335 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.529474 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.534482 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:06.534488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:06.534493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:06.534536 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:06.537798 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.539024 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.539032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.539039 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.539227 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.539897 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.539905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.539908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.540232 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:06.540404 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:06.541499 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0621 02:51:06.541727 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:51:06.543558 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.518809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.11.109.3:59081)
0621 02:51:06.519686 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:06.519737 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:06.521844 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:06.522988 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:06.523006 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:06.523910 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:06.524863 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.524883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.526700 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.526711 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.528478 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.528496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.529587 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.529599 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.529969 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.530804 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:06.530843 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:06.534261 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:06.537951 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.537967 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.537989 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.538824 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.539396 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.539401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.539406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.539739 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.540512 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:06.540524 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:06.540951 (+ 427us) client_negotiation.cc:769] Sending connection context
0621 02:51:06.541401 (+ 450us) client_negotiation.cc:240] Negotiation successful
0621 02:51:06.541990 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":387,"threads_started":1}
I20260621 02:51:06.559620 19636 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:44067
I20260621 02:51:06.559682 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:44067 every 8 connection(s)
I20260621 02:51:06.561317 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:51:06.563544 19771 heartbeater.cc:347] Connected to a master server at 127.11.109.62:36623
I20260621 02:51:06.563843 19771 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:06.564733 19771 heartbeater.cc:511] Master 127.11.109.62:36623 requested a full tablet report, sending...
I20260621 02:51:06.566413 19286 ts_manager.cc:194] Registered new tserver with Master: 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:06.566996 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 19636
I20260621 02:51:06.567191 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:51:06.567549 19286 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:59081
I20260621 02:51:06.567895 11700 external_mini_cluster.cc:1428] Setting key 7f49b571ea04bedd1141233b0d53219f
I20260621 02:51:06.568922 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.551180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.11.109.3:48181)
0621 02:51:06.553664 (+ 2484us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:06.553682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:06.553840 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:06.554259 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:06.554266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:06.555072 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:06.555627 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.555645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.557079 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.557085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.558445 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:06.558456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.559739 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.559747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:06.559956 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.560568 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:06.560585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:06.561465 (+ 880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:06.565475 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.565480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.565484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.566049 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.566934 (+ 885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:06.566942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:06.566946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:06.567091 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:06.567708 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:06.567712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:06.567920 (+ 208us) client_negotiation.cc:769] Sending connection context
0621 02:51:06.568050 (+ 130us) client_negotiation.cc:240] Negotiation successful
0621 02:51:06.568429 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2344,"thread_start_us":2204,"threads_started":1}
I20260621 02:51:06.570088 19493 heartbeater.cc:503] Master 127.11.109.62:36623 was elected leader, sending a full tablet report...
I20260621 02:51:06.570446 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:06.551608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:48181 (local address 127.11.109.62:36623)
0621 02:51:06.551842 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:06.551851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:06.553774 (+ 1923us) server_negotiation.cc:407] Connection header received
0621 02:51:06.553967 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:06.553974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:06.554076 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:06.554186 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:06.555801 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.556938 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.558609 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.559340 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.559477 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:06.561648 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:06.561654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:06.561661 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:06.561698 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:06.565314 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.566641 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.566646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.566649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.566785 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:06.567226 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:06.567231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:06.567234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:06.567564 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:06.569045 (+ 1481us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:06.569227 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 02:51:06.569602 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66}
I20260621 02:51:06.570951 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:51:06 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:51:06 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:06 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:07.087159 19632 heartbeater.cc:503] Master 127.11.109.62:36623 was elected leader, sending a full tablet report...
I20260621 02:51:07.571559 19771 heartbeater.cc:503] Master 127.11.109.62:36623 was elected leader, sending a full tablet report...
W20260621 02:51:07.666463 19779 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.052s user 0.005s sys 0.009s
W20260621 02:51:07.667718 19381 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:06.608512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.11.109.62:36623)
0621 02:51:06.608764 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:06.608770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:06.609013 (+ 243us) server_negotiation.cc:407] Connection header received
0621 02:51:06.609198 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:06.609204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:06.609300 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:06.609392 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:06.610647 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.611387 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.612554 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:06.612969 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:06.613060 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.667426 (+1054366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47922: BlockingRecv error: recv got EOF from 127.0.0.1:47922 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 21 02:51:07 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010266, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:07.693737 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.673264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.11.109.62:36623)
0621 02:51:07.673455 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.673462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.674082 (+ 620us) server_negotiation.cc:407] Connection header received
0621 02:51:07.674290 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.674297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.674379 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.674499 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:07.676385 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.677718 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.679909 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.680683 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.680827 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.686713 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:07.686722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:07.686727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:07.686774 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:07.690663 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:07.691553 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:07.691558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:07.691561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:07.691727 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:07.692127 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:07.692131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:07.692134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:07.692463 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:07.692747 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.693302 (+ 555us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.693507 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:51:07.699139 19286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:51:07.709139 19286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:07.748051 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.722966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39457 (local address 127.0.0.1:51290)
0621 02:51:07.732076 (+ 9110us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:07.732093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:07.733474 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:07.739073 (+ 5599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:07.739081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:07.739110 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:07.739567 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.739578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.741603 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.741608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.744024 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.744040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.747024 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.747030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.747214 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.747263 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 02:51:07.747457 (+ 194us) client_negotiation.cc:240] Negotiation successful
0621 02:51:07.747641 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8990,"thread_start_us":8837,"threads_started":1}
I20260621 02:51:07.749042 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.723550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.11.109.2:39457)
0621 02:51:07.737351 (+ 13801us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.737365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.737405 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:51:07.737526 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.737539 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.737906 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.740224 (+ 2318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:07.740368 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.741447 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.745338 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.746755 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.746931 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.747144 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.747544 (+ 400us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.747757 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13553,"thread_start_us":14540,"threads_started":1}
I20260621 02:51:07.749260 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.722311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:37613 (local address 127.0.0.1:60004)
0621 02:51:07.729640 (+ 7329us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:07.729697 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:07.729908 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:07.742234 (+ 12326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:07.742239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:07.742271 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:07.742749 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.742761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.744839 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.744843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.746614 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.746628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.748338 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.748346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.748538 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.748578 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:51:07.748643 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 02:51:07.748773 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":23,"thread_start_us":287,"threads_started":1}
I20260621 02:51:07.749915 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.722560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.11.109.1:37613)
0621 02:51:07.741666 (+ 19106us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.741676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.741715 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:51:07.741819 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.741828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.742048 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.742241 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:07.743125 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.744701 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.746771 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.748099 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.749224 (+ 1125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.749433 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.749498 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.749640 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18938,"thread_start_us":18791,"threads_started":1}
I20260621 02:51:07.750123 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.723765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.11.109.3:44067)
0621 02:51:07.737528 (+ 13763us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.737541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.738890 (+ 1349us) server_negotiation.cc:407] Connection header received
0621 02:51:07.739019 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.739029 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.739318 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.739544 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:07.740178 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.741892 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.746365 (+ 4473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.747931 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.748071 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.749662 (+ 1591us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.749713 (+ 51us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.749835 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13335,"thread_start_us":13393,"threads_started":1}
I20260621 02:51:07.750391 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.723412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:44067 (local address 127.0.0.1:34486)
0621 02:51:07.738728 (+ 15316us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:07.738768 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:07.738887 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:07.739734 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:07.739738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:07.739763 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:07.740057 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.740065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.744175 (+ 4110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.744188 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.746161 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.746174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.748446 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.748452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.748605 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.748640 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:51:07.748718 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 02:51:07.748819 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15206,"mutex_wait_us":1,"thread_start_us":15077,"threads_started":1}
I20260621 02:51:07.752805 19423 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7e08d5116944a2949180b7e37ca083 (DEFAULT_TABLE table=test-table [id=aed3e89bced14b17bae9d85f0170306b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:07.752842 19562 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7e08d5116944a2949180b7e37ca083 (DEFAULT_TABLE table=test-table [id=aed3e89bced14b17bae9d85f0170306b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:07.753548 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7e08d5116944a2949180b7e37ca083 (DEFAULT_TABLE table=test-table [id=aed3e89bced14b17bae9d85f0170306b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:07.754204 19423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7e08d5116944a2949180b7e37ca083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:07.754393 19562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7e08d5116944a2949180b7e37ca083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:07.754760 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7e08d5116944a2949180b7e37ca083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:07.764961 19796 tablet_bootstrap.cc:492] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Bootstrap starting.
I20260621 02:51:07.768042 19796 tablet_bootstrap.cc:654] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:07.768404 19797 tablet_bootstrap.cc:492] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Bootstrap starting.
I20260621 02:51:07.769796 19798 tablet_bootstrap.cc:492] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Bootstrap starting.
I20260621 02:51:07.771477 19796 log.cc:824] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:07.772009 19797 tablet_bootstrap.cc:654] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:07.772886 19798 tablet_bootstrap.cc:654] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:07.773092 19797 log.cc:824] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:07.774042 19798 log.cc:824] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:07.774479 19796 tablet_bootstrap.cc:492] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: No bootstrap required, opened a new log
I20260621 02:51:07.774785 19796 ts_tablet_manager.cc:1397] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:51:07.775950 19797 tablet_bootstrap.cc:492] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: No bootstrap required, opened a new log
I20260621 02:51:07.776242 19797 ts_tablet_manager.cc:1397] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260621 02:51:07.776481 19798 tablet_bootstrap.cc:492] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: No bootstrap required, opened a new log
I20260621 02:51:07.776773 19798 ts_tablet_manager.cc:1397] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260621 02:51:07.786439 19798 raft_consensus.cc:348] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.786573 19797 raft_consensus.cc:348] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.786972 19798 raft_consensus.cc:374] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:07.787081 19797 raft_consensus.cc:374] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:07.787103 19798 raft_consensus.cc:729] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed257eaeefc45a38cd4b2845db2a47c, State: Initialized, Role: FOLLOWER
I20260621 02:51:07.787215 19797 raft_consensus.cc:729] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Initialized, Role: FOLLOWER
I20260621 02:51:07.787281 19796 raft_consensus.cc:348] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.787688 19796 raft_consensus.cc:374] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:07.787801 19796 raft_consensus.cc:729] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59d2e132704b7bbe3c5dc5c1e2e117, State: Initialized, Role: FOLLOWER
I20260621 02:51:07.787868 19797 consensus_queue.cc:260] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.787958 19798 consensus_queue.cc:260] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.788496 19796 consensus_queue.cc:260] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.796387 19797 ts_tablet_manager.cc:1428] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
I20260621 02:51:07.797962 19798 ts_tablet_manager.cc:1428] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent starting tablet: real 0.021s user 0.022s sys 0.000s
I20260621 02:51:07.800531 19796 ts_tablet_manager.cc:1428] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent starting tablet: real 0.026s user 0.007s sys 0.014s
W20260621 02:51:07.814167 19494 tablet.cc:2369] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:51:07.819730 19772 tablet.cc:2369] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:51:07.832414 19633 tablet.cc:2369] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:51:07.929625 19802 raft_consensus.cc:482] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:07.929997 19802 raft_consensus.cc:504] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.931690 19802 leader_election.cc:288] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:07.940186 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.933014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:50677 (local address 127.11.109.3:44067)
0621 02:51:07.933236 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.933241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.933445 (+ 204us) server_negotiation.cc:407] Connection header received
0621 02:51:07.933594 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.933597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.933660 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.933728 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:07.934267 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.935214 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.937753 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.939016 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.939172 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.939244 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.939807 (+ 563us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.939939 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:51:07.940186 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.932151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:36047 (local address 127.11.109.1:37613)
0621 02:51:07.932336 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.932340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.932748 (+ 408us) server_negotiation.cc:407] Connection header received
0621 02:51:07.932956 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.932961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.933045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.933147 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:07.933929 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.934948 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.937763 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.939015 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.939173 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.939244 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.939807 (+ 563us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.939939 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:51:07.940430 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.931984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:37613 (local address 127.11.109.2:36047)
0621 02:51:07.932628 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:07.932648 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:07.932801 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:07.933288 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:07.933293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:07.933317 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:07.933760 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.933771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.935087 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.935090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.937624 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.937633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.939317 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.939323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.939595 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.939674 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 02:51:07.939770 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:51:07.939931 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":221,"spinlock_wait_cycles":23296,"thread_start_us":329,"threads_started":1}
I20260621 02:51:07.940465 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.932774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:44067 (local address 127.11.109.2:50677)
0621 02:51:07.933339 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:07.933359 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:07.933490 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:07.933776 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:07.933779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:07.933804 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:07.934127 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.934134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.935346 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.935352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.937601 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:07.937620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.939292 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.939304 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:07.939603 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.939686 (+ 83us) client_negotiation.cc:769] Sending connection context
0621 02:51:07.939770 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 02:51:07.939928 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":151,"thread_start_us":287,"threads_started":1}
I20260621 02:51:07.941502 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7e08d5116944a2949180b7e37ca083" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" is_pre_election: true
I20260621 02:51:07.941502 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7e08d5116944a2949180b7e37ca083" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" is_pre_election: true
I20260621 02:51:07.942061 19447 raft_consensus.cc:2454] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 0.
I20260621 02:51:07.942054 19721 raft_consensus.cc:2454] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 0.
I20260621 02:51:07.942704 19515 leader_election.cc:302] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:07.943039 19802 raft_consensus.cc:2781] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:07.943212 19802 raft_consensus.cc:482] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:07.943322 19802 raft_consensus.cc:3037] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:07.945503 19802 raft_consensus.cc:504] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.946074 19802 leader_election.cc:288] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:07.946599 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7e08d5116944a2949180b7e37ca083" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
I20260621 02:51:07.946630 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7e08d5116944a2949180b7e37ca083" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
I20260621 02:51:07.946810 19721 raft_consensus.cc:3037] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:07.946810 19447 raft_consensus.cc:3037] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:07.950029 19721 raft_consensus.cc:2454] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 1.
I20260621 02:51:07.950029 19447 raft_consensus.cc:2454] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 1.
I20260621 02:51:07.950403 19515 leader_election.cc:302] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:07.950717 19802 raft_consensus.cc:2781] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:07.951161 19802 raft_consensus.cc:686] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Running, Role: LEADER
I20260621 02:51:07.951624 19802 consensus_queue.cc:237] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } }
I20260621 02:51:07.956224 19284 catalog_manager.cc:5495] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2). New cstate: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } health_report { overall_health: HEALTHY } } }
I20260621 02:51:07.982856 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:07.977482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.11.109.2:39457)
0621 02:51:07.977702 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:07.977709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:07.977805 (+ 96us) server_negotiation.cc:407] Connection header received
0621 02:51:07.978076 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:07.978080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:07.978172 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:07.978322 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:07.978672 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.979405 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.980716 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:07.981118 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:07.981342 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:07.981785 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:07.982295 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:07.982459 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:07.982517 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 02:51:07.982595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:51:07.990566 19721 raft_consensus.cc:1261] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Refusing update from remote peer 37ab78a2e85943b9820cdbf29ec3f4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:07.990737 19447 raft_consensus.cc:1261] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Refusing update from remote peer 37ab78a2e85943b9820cdbf29ec3f4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:07.991539 19807 consensus_queue.cc:1035] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:07.991951 19802 consensus_queue.cc:1035] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.002389 19810 mvcc.cc:205] Tried to move back new op lower bound from 7299114057679626240 to 7299114057533501440. Current Snapshot: MvccSnapshot[applied={T|T < 7299114057679626240}]
I20260621 02:51:08.015196 19284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:08.015478 19284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:08.018716 19284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Sending DeleteTablet for 3 replicas of tablet 9e7e08d5116944a2949180b7e37ca083
I20260621 02:51:08.019812 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7e08d5116944a2949180b7e37ca083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:34486
I20260621 02:51:08.019793 19562 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7e08d5116944a2949180b7e37ca083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260621 02:51:08.020551 19423 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7e08d5116944a2949180b7e37ca083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60004
I20260621 02:51:08.020711 19820 tablet_replica.cc:330] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: stopping tablet replica
I20260621 02:51:08.021112 19820 raft_consensus.cc:2229] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:51:08.021065 19284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 02:51:08.022086 19820 raft_consensus.cc:2258] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260621 02:51:08.022257 19284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:08.024667 19820 ts_tablet_manager.cc:1905] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.030505 19821 tablet_replica.cc:330] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: stopping tablet replica
I20260621 02:51:08.030925 19821 raft_consensus.cc:2229] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:08.031447 19821 raft_consensus.cc:2258] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.031778 19422 tablet_service.cc:1467] Processing CreateTablet for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf (DEFAULT_TABLE table=test-table [id=23f4874d51d0466eb0d7713e28add27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:08.032320 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.032358 19819 tablet_replica.cc:330] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: stopping tablet replica
I20260621 02:51:08.032745 19819 raft_consensus.cc:2229] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:08.033197 19819 raft_consensus.cc:2258] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.034761 19821 ts_tablet_manager.cc:1905] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.038053 19562 tablet_service.cc:1467] Processing CreateTablet for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf (DEFAULT_TABLE table=test-table [id=23f4874d51d0466eb0d7713e28add27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:08.038319 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf (DEFAULT_TABLE table=test-table [id=23f4874d51d0466eb0d7713e28add27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:08.038453 19562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.038734 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.039027 19796 tablet_bootstrap.cc:492] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Bootstrap starting.
I20260621 02:51:08.040570 19796 tablet_bootstrap.cc:654] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.044802 19798 tablet_bootstrap.cc:492] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Bootstrap starting.
I20260621 02:51:08.045245 19820 ts_tablet_manager.cc:1918] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:08.045511 19820 log.cc:1196] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/9e7e08d5116944a2949180b7e37ca083
I20260621 02:51:08.045976 19797 tablet_bootstrap.cc:492] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Bootstrap starting.
I20260621 02:51:08.046219 19820 ts_tablet_manager.cc:1939] T 9e7e08d5116944a2949180b7e37ca083 P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting consensus metadata
I20260621 02:51:08.046517 19798 tablet_bootstrap.cc:654] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.046838 19819 ts_tablet_manager.cc:1905] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.047660 19797 tablet_bootstrap.cc:654] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.049450 19273 catalog_manager.cc:4841] TS 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457): tablet 9e7e08d5116944a2949180b7e37ca083 (table test-table [id=aed3e89bced14b17bae9d85f0170306b]) successfully deleted
I20260621 02:51:08.059342 19797 tablet_bootstrap.cc:492] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: No bootstrap required, opened a new log
I20260621 02:51:08.059314 19821 ts_tablet_manager.cc:1918] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:08.059342 19796 tablet_bootstrap.cc:492] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: No bootstrap required, opened a new log
I20260621 02:51:08.059499 19821 log.cc:1196] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/9e7e08d5116944a2949180b7e37ca083
I20260621 02:51:08.059516 19797 ts_tablet_manager.cc:1397] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20260621 02:51:08.059537 19796 ts_tablet_manager.cc:1397] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20260621 02:51:08.060075 19821 ts_tablet_manager.cc:1939] T 9e7e08d5116944a2949180b7e37ca083 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting consensus metadata
I20260621 02:51:08.060439 19796 raft_consensus.cc:348] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.060442 19797 raft_consensus.cc:348] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.060686 19796 raft_consensus.cc:374] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.060676 19797 raft_consensus.cc:374] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.060791 19796 raft_consensus.cc:729] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59d2e132704b7bbe3c5dc5c1e2e117, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.060791 19797 raft_consensus.cc:729] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.061076 19797 consensus_queue.cc:260] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.061110 19796 consensus_queue.cc:260] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.061745 19797 ts_tablet_manager.cc:1428] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:51:08.062249 19798 tablet_bootstrap.cc:492] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: No bootstrap required, opened a new log
I20260621 02:51:08.062422 19798 ts_tablet_manager.cc:1397] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20260621 02:51:08.062565 19272 catalog_manager.cc:4841] TS 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613): tablet 9e7e08d5116944a2949180b7e37ca083 (table test-table [id=aed3e89bced14b17bae9d85f0170306b]) successfully deleted
I20260621 02:51:08.063446 19819 ts_tablet_manager.cc:1918] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:08.063644 19819 log.cc:1196] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/9e7e08d5116944a2949180b7e37ca083
I20260621 02:51:08.063673 19796 ts_tablet_manager.cc:1428] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:51:08.063827 19798 raft_consensus.cc:348] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.064057 19798 raft_consensus.cc:374] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.064157 19798 raft_consensus.cc:729] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed257eaeefc45a38cd4b2845db2a47c, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.064244 19819 ts_tablet_manager.cc:1939] T 9e7e08d5116944a2949180b7e37ca083 P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting consensus metadata
I20260621 02:51:08.064507 19798 consensus_queue.cc:260] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.065027 19798 ts_tablet_manager.cc:1428] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 02:51:08.065887 19272 catalog_manager.cc:4841] TS 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067): tablet 9e7e08d5116944a2949180b7e37ca083 (table test-table [id=aed3e89bced14b17bae9d85f0170306b]) successfully deleted
I20260621 02:51:08.304998 19807 raft_consensus.cc:482] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:08.305258 19807 raft_consensus.cc:504] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.305989 19807 leader_election.cc:288] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:08.306560 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d5c8198817412b9cdb3fbbdf8e4bcf" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" is_pre_election: true
I20260621 02:51:08.306648 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d5c8198817412b9cdb3fbbdf8e4bcf" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" is_pre_election: true
I20260621 02:51:08.306813 19447 raft_consensus.cc:2454] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 0.
I20260621 02:51:08.306939 19721 raft_consensus.cc:2454] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 0.
I20260621 02:51:08.307288 19515 leader_election.cc:302] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:08.307598 19807 raft_consensus.cc:2781] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:08.307706 19807 raft_consensus.cc:482] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:08.307847 19807 raft_consensus.cc:3037] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.310091 19807 raft_consensus.cc:504] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.310676 19807 leader_election.cc:288] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:08.311126 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d5c8198817412b9cdb3fbbdf8e4bcf" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
I20260621 02:51:08.311232 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d5c8198817412b9cdb3fbbdf8e4bcf" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
I20260621 02:51:08.311311 19447 raft_consensus.cc:3037] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.311446 19721 raft_consensus.cc:3037] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.313354 19447 raft_consensus.cc:2454] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 1.
I20260621 02:51:08.313750 19515 leader_election.cc:302] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:08.313989 19807 raft_consensus.cc:2781] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:08.314178 19807 raft_consensus.cc:686] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Running, Role: LEADER
I20260621 02:51:08.314427 19807 consensus_queue.cc:237] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.314818 19721 raft_consensus.cc:2454] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 1.
I20260621 02:51:08.317478 19285 catalog_manager.cc:5495] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2). New cstate: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } health_report { overall_health: UNKNOWN } } }
Jun 21 02:51:08 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010264, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.62@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:08.390232 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.375488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:36623 (local address 127.0.0.1:47942)
0621 02:51:08.376435 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:08.376449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:08.376639 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:08.377004 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:08.377009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:08.377980 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:08.378266 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.378280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.379312 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.379318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.380784 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.380798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.381604 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.381611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.381798 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.382483 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:08.382509 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:08.385024 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:08.387892 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:08.387905 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:08.387912 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:08.388276 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:08.388593 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:08.388598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:08.388601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:08.388746 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:08.389236 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:08.389244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:08.389393 (+ 149us) client_negotiation.cc:769] Sending connection context
0621 02:51:08.389552 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 02:51:08.389806 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":684,"threads_started":1}
I20260621 02:51:08.390252 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.375476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.11.109.62:36623)
0621 02:51:08.376276 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.376283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.376551 (+ 268us) server_negotiation.cc:407] Connection header received
0621 02:51:08.376755 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.376760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.376819 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.376949 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:08.378425 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.379177 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.380914 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.381435 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.381527 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.385233 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:08.385238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:08.385241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:08.385268 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:08.387748 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:08.388388 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:08.388394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:08.388399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:08.388499 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:08.388844 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:08.388848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:08.388851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:08.389097 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:08.389260 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.389620 (+ 360us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.389917 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":635,"thread_start_us":435,"threads_started":1}
I20260621 02:51:08.395552 19285 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:47942:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:51:08.396757 19285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:08.403343 19422 tablet_service.cc:1467] Processing CreateTablet for tablet 33a21a3756154cb681855d34050fc25a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c99f424987ae432c9c9017037e45fce5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:51:08.403877 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a21a3756154cb681855d34050fc25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.409919 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 33a21a3756154cb681855d34050fc25a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c99f424987ae432c9c9017037e45fce5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:51:08.409920 19562 tablet_service.cc:1467] Processing CreateTablet for tablet 33a21a3756154cb681855d34050fc25a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c99f424987ae432c9c9017037e45fce5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:51:08.410473 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a21a3756154cb681855d34050fc25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.410480 19562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a21a3756154cb681855d34050fc25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.411228 19796 tablet_bootstrap.cc:492] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: Bootstrap starting.
I20260621 02:51:08.412875 19796 tablet_bootstrap.cc:654] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.417699 19798 tablet_bootstrap.cc:492] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: Bootstrap starting.
I20260621 02:51:08.419893 19798 tablet_bootstrap.cc:654] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.420106 19797 tablet_bootstrap.cc:492] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: Bootstrap starting.
I20260621 02:51:08.421864 19797 tablet_bootstrap.cc:654] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.422531 19798 tablet_bootstrap.cc:492] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: No bootstrap required, opened a new log
I20260621 02:51:08.422670 19798 ts_tablet_manager.cc:1397] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260621 02:51:08.423635 19798 raft_consensus.cc:348] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.423939 19798 raft_consensus.cc:374] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.424023 19798 raft_consensus.cc:729] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed257eaeefc45a38cd4b2845db2a47c, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.424851 19796 tablet_bootstrap.cc:492] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: No bootstrap required, opened a new log
I20260621 02:51:08.425020 19796 ts_tablet_manager.cc:1397] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260621 02:51:08.425593 19798 consensus_queue.cc:260] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.425796 19796 raft_consensus.cc:348] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.426046 19796 raft_consensus.cc:374] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.426144 19796 raft_consensus.cc:729] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59d2e132704b7bbe3c5dc5c1e2e117, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.426162 19803 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.426354 19796 consensus_queue.cc:260] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.426548 19803 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.426998 19796 ts_tablet_manager.cc:1428] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260621 02:51:08.426848 19804 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.427227 19804 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.427948 19798 ts_tablet_manager.cc:1428] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260621 02:51:08.429662 19797 tablet_bootstrap.cc:492] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: No bootstrap required, opened a new log
I20260621 02:51:08.429903 19797 ts_tablet_manager.cc:1397] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:51:08.430721 19797 raft_consensus.cc:348] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.430907 19797 raft_consensus.cc:374] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.431034 19797 raft_consensus.cc:729] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.431288 19797 consensus_queue.cc:260] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.431740 19807 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.431886 19797 ts_tablet_manager.cc:1428] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 02:51:08.431993 19807 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.490648 19807 raft_consensus.cc:482] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:08.490813 19807 raft_consensus.cc:504] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.491351 19807 leader_election.cc:288] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:08.491835 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a21a3756154cb681855d34050fc25a" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" is_pre_election: true
I20260621 02:51:08.492112 19447 raft_consensus.cc:2454] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 0.
I20260621 02:51:08.491966 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a21a3756154cb681855d34050fc25a" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" is_pre_election: true
I20260621 02:51:08.492322 19721 raft_consensus.cc:2454] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 0.
I20260621 02:51:08.492512 19515 leader_election.cc:302] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:08.492789 19807 raft_consensus.cc:2781] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:08.492924 19807 raft_consensus.cc:482] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:08.493031 19807 raft_consensus.cc:3037] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.494999 19807 raft_consensus.cc:504] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.495465 19807 leader_election.cc:288] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 election: Requested vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:08.495859 19447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a21a3756154cb681855d34050fc25a" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
I20260621 02:51:08.495942 19721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a21a3756154cb681855d34050fc25a" candidate_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
I20260621 02:51:08.496042 19447 raft_consensus.cc:3037] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.496194 19721 raft_consensus.cc:3037] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.497973 19447 raft_consensus.cc:2454] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 1.
I20260621 02:51:08.498147 19721 raft_consensus.cc:2454] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab78a2e85943b9820cdbf29ec3f4b3 in term 1.
I20260621 02:51:08.498529 19515 leader_election.cc:302] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ab78a2e85943b9820cdbf29ec3f4b3, 7ed257eaeefc45a38cd4b2845db2a47c; no voters:
I20260621 02:51:08.498816 19807 raft_consensus.cc:2781] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:08.498977 19807 raft_consensus.cc:686] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Running, Role: LEADER
I20260621 02:51:08.499300 19807 consensus_queue.cc:237] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.500665 19802 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: TxnStatusTablet state changed. Reason: New leader 37ab78a2e85943b9820cdbf29ec3f4b3. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.500947 19802 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:51:08.501578 19837 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:51:08.501791 19837 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:51:08.502250 19284 catalog_manager.cc:5495] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2). New cstate: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } health_report { overall_health: UNKNOWN } } }
I20260621 02:51:08.521885 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.516417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39457 (local address 127.0.0.1:51310)
0621 02:51:08.516714 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:08.516737 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:08.516876 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:08.517349 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:08.517353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:08.517363 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:51:08.517498 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.517504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.518929 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.518932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.519875 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.519886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.520819 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.520822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.520963 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.521009 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:51:08.521485 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:51:08.521500 (+ 15us) client_negotiation.cc:769] Sending connection context
0621 02:51:08.521566 (+ 66us) client_negotiation.cc:240] Negotiation successful
0621 02:51:08.521670 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 02:51:08.522033 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.516364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.11.109.2:39457)
0621 02:51:08.517005 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.517011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.517046 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:51:08.517106 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.517111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.517208 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.517318 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:08.517631 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.518811 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.520008 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.520676 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.520809 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.521240 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:08.521389 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:08.521489 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.521627 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.521707 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":394,"threads_started":1}
I20260621 02:51:08.525802 19721 raft_consensus.cc:1261] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Refusing update from remote peer 37ab78a2e85943b9820cdbf29ec3f4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:08.525918 19447 raft_consensus.cc:1261] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Refusing update from remote peer 37ab78a2e85943b9820cdbf29ec3f4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:08.526489 19807 consensus_queue.cc:1035] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.527058 19802 consensus_queue.cc:1035] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.534371 19804 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: TxnStatusTablet state changed. Reason: New leader 37ab78a2e85943b9820cdbf29ec3f4b3. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.534595 19804 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.535085 19803 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: TxnStatusTablet state changed. Reason: New leader 37ab78a2e85943b9820cdbf29ec3f4b3. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.535337 19803 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.544870 19807 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.545238 19807 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:51:08.545744 19803 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.546020 19803 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 7ed257eaeefc45a38cd4b2845db2a47c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.550246 19807 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.550745 19807 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 37ab78a2e85943b9820cdbf29ec3f4b3: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:51:08.554217 19804 tablet_replica.cc:439] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } } }
I20260621 02:51:08.554471 19804 tablet_replica.cc:442] T 33a21a3756154cb681855d34050fc25a P 0d59d2e132704b7bbe3c5dc5c1e2e117: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:51:08.605526 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.598969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:41935 (local address 127.11.109.2:39457)
0621 02:51:08.599136 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.599142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.599787 (+ 645us) server_negotiation.cc:407] Connection header received
0621 02:51:08.599999 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.600005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.600082 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.600180 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:08.600744 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.601957 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.603865 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.604251 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.604393 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.604794 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:08.604884 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:08.604985 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.605111 (+ 126us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.605192 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260621 02:51:08.605597 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.598994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39457 (local address 127.11.109.2:41935)
0621 02:51:08.599675 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:08.599698 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:08.599861 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:08.600210 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:08.600215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:08.600227 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:51:08.600566 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.600578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.602124 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.602129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.603727 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.603738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.604416 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.604421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.604568 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.604615 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:51:08.604983 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:51:08.604993 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:51:08.605079 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:51:08.605233 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":335,"threads_started":1}
I20260621 02:51:08.614311 19721 raft_consensus.cc:1261] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Refusing update from remote peer 37ab78a2e85943b9820cdbf29ec3f4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:08.614424 19447 raft_consensus.cc:1261] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Refusing update from remote peer 37ab78a2e85943b9820cdbf29ec3f4b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:08.615038 19849 consensus_queue.cc:1035] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.615572 19807 consensus_queue.cc:1035] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.626036 19842 mvcc.cc:205] Tried to move back new op lower bound from 7299114060240076800 to 7299114059018551296. Current Snapshot: MvccSnapshot[applied={T|T < 7299114060240076800}]
I20260621 02:51:08.675005 19285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:08.675256 19285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:08.678193 19285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Sending DeleteTablet for 3 replicas of tablet 83d5c8198817412b9cdb3fbbdf8e4bcf
I20260621 02:51:08.678918 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:34486
I20260621 02:51:08.678969 19422 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60004
I20260621 02:51:08.689652 19561 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d5c8198817412b9cdb3fbbdf8e4bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260621 02:51:08.691322 19856 tablet_replica.cc:330] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: stopping tablet replica
I20260621 02:51:08.691682 19856 raft_consensus.cc:2229] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:08.691982 19856 raft_consensus.cc:2258] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.693228 19856 ts_tablet_manager.cc:1905] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.700739 19856 ts_tablet_manager.cc:1918] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:51:08.700982 19856 log.cc:1196] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/83d5c8198817412b9cdb3fbbdf8e4bcf
I20260621 02:51:08.701918 19856 ts_tablet_manager.cc:1939] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting consensus metadata
I20260621 02:51:08.702076 19858 tablet_replica.cc:330] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: stopping tablet replica
I20260621 02:51:08.702523 19858 raft_consensus.cc:2229] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:51:08.703256 19858 raft_consensus.cc:2258] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.703248 19272 catalog_manager.cc:4841] TS 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067): tablet 83d5c8198817412b9cdb3fbbdf8e4bcf (table test-table [id=23f4874d51d0466eb0d7713e28add27a]) successfully deleted
I20260621 02:51:08.703953 19857 tablet_replica.cc:330] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: stopping tablet replica
I20260621 02:51:08.704197 19857 raft_consensus.cc:2229] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:08.704490 19857 raft_consensus.cc:2258] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.704519 19858 ts_tablet_manager.cc:1905] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.705683 19857 ts_tablet_manager.cc:1905] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.711990 19858 ts_tablet_manager.cc:1918] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:51:08.712141 19858 log.cc:1196] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/83d5c8198817412b9cdb3fbbdf8e4bcf
I20260621 02:51:08.712360 19857 ts_tablet_manager.cc:1918] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:51:08.712539 19857 log.cc:1196] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/83d5c8198817412b9cdb3fbbdf8e4bcf
I20260621 02:51:08.712733 19858 ts_tablet_manager.cc:1939] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting consensus metadata
I20260621 02:51:08.713042 19857 ts_tablet_manager.cc:1939] T 83d5c8198817412b9cdb3fbbdf8e4bcf P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting consensus metadata
I20260621 02:51:08.713955 19273 catalog_manager.cc:4841] TS 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457): tablet 83d5c8198817412b9cdb3fbbdf8e4bcf (table test-table [id=23f4874d51d0466eb0d7713e28add27a]) successfully deleted
I20260621 02:51:08.714357 19272 catalog_manager.cc:4841] TS 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613): tablet 83d5c8198817412b9cdb3fbbdf8e4bcf (table test-table [id=23f4874d51d0466eb0d7713e28add27a]) successfully deleted
Jun 21 02:51:08 dist-test-slave-bc52 krb5kdc[19241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:51:08 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:08 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:51:08.758040 19831 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:08.747507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.11.109.62:36623)
0621 02:51:08.747721 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.747730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.748065 (+ 335us) server_negotiation.cc:407] Connection header received
0621 02:51:08.748317 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.748322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.748432 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.748551 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:08.749866 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.750560 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.751774 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.752209 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.752352 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.757796 (+ 5444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47954: BlockingRecv error: recv got EOF from 127.0.0.1:47954 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 02:51:08 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM
I20260621 02:51:08.775820 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.760758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.11.109.62:36623)
0621 02:51:08.760923 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.760927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.761476 (+ 549us) server_negotiation.cc:407] Connection header received
0621 02:51:08.761627 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.761632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.761714 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.761823 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:08.762899 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.764034 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.765352 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.765994 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.766121 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.769716 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:08.769723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:08.769727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:08.769766 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:08.772959 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:08.773858 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:08.773864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:08.773868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:08.774040 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:08.774432 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:08.774437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:08.774440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:08.774767 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:08.775034 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.775231 (+ 197us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.775475 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:51:08.779546 19285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:51:08.780230 19285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:08.786932 19422 tablet_service.cc:1467] Processing CreateTablet for tablet 50cd87393d5f449494bb0330a537f8da (DEFAULT_TABLE table=test-table [id=767ce0f84a464238a7d12c25776bbfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:08.787395 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cd87393d5f449494bb0330a537f8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.787346 19561 tablet_service.cc:1467] Processing CreateTablet for tablet 50cd87393d5f449494bb0330a537f8da (DEFAULT_TABLE table=test-table [id=767ce0f84a464238a7d12c25776bbfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:08.787358 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 50cd87393d5f449494bb0330a537f8da (DEFAULT_TABLE table=test-table [id=767ce0f84a464238a7d12c25776bbfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:08.787770 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cd87393d5f449494bb0330a537f8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.787829 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50cd87393d5f449494bb0330a537f8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:08.794019 19796 tablet_bootstrap.cc:492] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Bootstrap starting.
I20260621 02:51:08.794682 19797 tablet_bootstrap.cc:492] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Bootstrap starting.
I20260621 02:51:08.794803 19798 tablet_bootstrap.cc:492] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Bootstrap starting.
I20260621 02:51:08.795617 19796 tablet_bootstrap.cc:654] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.796299 19798 tablet_bootstrap.cc:654] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.796373 19797 tablet_bootstrap.cc:654] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:08.806810 19796 tablet_bootstrap.cc:492] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: No bootstrap required, opened a new log
I20260621 02:51:08.806977 19796 ts_tablet_manager.cc:1397] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260621 02:51:08.807276 19798 tablet_bootstrap.cc:492] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: No bootstrap required, opened a new log
I20260621 02:51:08.807477 19798 ts_tablet_manager.cc:1397] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260621 02:51:08.807611 19797 tablet_bootstrap.cc:492] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: No bootstrap required, opened a new log
I20260621 02:51:08.807753 19797 ts_tablet_manager.cc:1397] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260621 02:51:08.807746 19796 raft_consensus.cc:348] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.808022 19796 raft_consensus.cc:374] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.808115 19796 raft_consensus.cc:729] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59d2e132704b7bbe3c5dc5c1e2e117, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.808375 19796 consensus_queue.cc:260] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.808760 19798 raft_consensus.cc:348] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.808918 19796 ts_tablet_manager.cc:1428] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:51:08.809052 19798 raft_consensus.cc:374] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.808568 19797 raft_consensus.cc:348] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.809267 19798 raft_consensus.cc:729] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed257eaeefc45a38cd4b2845db2a47c, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.809365 19797 raft_consensus.cc:374] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:08.809474 19797 raft_consensus.cc:729] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Initialized, Role: FOLLOWER
I20260621 02:51:08.809690 19797 consensus_queue.cc:260] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.809685 19798 consensus_queue.cc:260] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.810230 19797 ts_tablet_manager.cc:1428] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:51:08.811211 19798 ts_tablet_manager.cc:1428] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260621 02:51:08.922266 19803 raft_consensus.cc:482] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:08.922510 19803 raft_consensus.cc:504] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.924351 19803 leader_election.cc:288] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457)
I20260621 02:51:08.933804 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.924915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54921 (local address 127.11.109.1:37613)
0621 02:51:08.925656 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.925665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.926568 (+ 903us) server_negotiation.cc:407] Connection header received
0621 02:51:08.926840 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.926845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.926918 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.926984 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:08.928060 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.928900 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.931835 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.932721 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.932932 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.933005 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.933313 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.933455 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":454,"threads_started":1}
I20260621 02:51:08.934062 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.924660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39457 (local address 127.11.109.3:50185)
0621 02:51:08.926143 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:08.926164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:08.926647 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:08.927186 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:08.927192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:08.927241 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:08.927611 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.927627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.929349 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.929355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.931656 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.931671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.933297 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.933303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.933459 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.933486 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 02:51:08.933578 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:51:08.933693 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"mutex_wait_us":96,"spinlock_wait_cycles":41088,"thread_start_us":1159,"threads_started":1}
I20260621 02:51:08.934106 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.924849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:50185 (local address 127.11.109.2:39457)
0621 02:51:08.925093 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.925103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.926300 (+ 1197us) server_negotiation.cc:407] Connection header received
0621 02:51:08.926839 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.926845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.926918 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.926982 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:08.927857 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.929215 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.931811 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.933060 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.933240 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.933306 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.933606 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.933734 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:51:08.934760 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.924664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:37613 (local address 127.11.109.3:54921)
0621 02:51:08.926467 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:08.926482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:08.926647 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:08.927212 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:08.927217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:08.927431 (+ 214us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:08.927878 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.927886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.929024 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.929030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.931656 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:08.931671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.932919 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.932925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:08.933087 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.933108 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:51:08.934277 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0621 02:51:08.934405 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"mutex_wait_us":547,"spinlock_wait_cycles":38656,"thread_start_us":1069,"threads_started":1}
I20260621 02:51:08.934929 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cd87393d5f449494bb0330a537f8da" candidate_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" is_pre_election: true
I20260621 02:51:08.935336 19587 raft_consensus.cc:2454] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed257eaeefc45a38cd4b2845db2a47c in term 0.
I20260621 02:51:08.935627 19446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cd87393d5f449494bb0330a537f8da" candidate_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" is_pre_election: true
I20260621 02:51:08.935895 19446 raft_consensus.cc:2454] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed257eaeefc45a38cd4b2845db2a47c in term 0.
I20260621 02:51:08.935920 19655 leader_election.cc:302] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ab78a2e85943b9820cdbf29ec3f4b3, 7ed257eaeefc45a38cd4b2845db2a47c; no voters:
I20260621 02:51:08.936586 19803 raft_consensus.cc:2781] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:08.936692 19803 raft_consensus.cc:482] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:08.936751 19803 raft_consensus.cc:3037] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.938882 19803 raft_consensus.cc:504] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.939422 19803 leader_election.cc:288] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [CANDIDATE]: Term 1 election: Requested vote from peers 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613), 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457)
I20260621 02:51:08.939878 19446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cd87393d5f449494bb0330a537f8da" candidate_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117"
I20260621 02:51:08.940064 19446 raft_consensus.cc:3037] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.939996 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50cd87393d5f449494bb0330a537f8da" candidate_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3"
I20260621 02:51:08.940222 19587 raft_consensus.cc:3037] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:08.942396 19587 raft_consensus.cc:2454] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed257eaeefc45a38cd4b2845db2a47c in term 1.
I20260621 02:51:08.942747 19655 leader_election.cc:302] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ab78a2e85943b9820cdbf29ec3f4b3, 7ed257eaeefc45a38cd4b2845db2a47c; no voters:
I20260621 02:51:08.942880 19446 raft_consensus.cc:2454] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed257eaeefc45a38cd4b2845db2a47c in term 1.
I20260621 02:51:08.943008 19803 raft_consensus.cc:2781] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:08.943452 19803 raft_consensus.cc:686] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 LEADER]: Becoming Leader. State: Replica: 7ed257eaeefc45a38cd4b2845db2a47c, State: Running, Role: LEADER
I20260621 02:51:08.943822 19803 consensus_queue.cc:237] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:08.947261 19284 catalog_manager.cc:5495] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3). New cstate: current_term: 1 leader_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } health_report { overall_health: HEALTHY } } }
I20260621 02:51:08.968035 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:08.961834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.11.109.3:44067)
0621 02:51:08.962508 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:08.962512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:08.962529 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:51:08.962600 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:08.962603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:08.962673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:08.962811 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:08.963177 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.963921 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.965835 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:08.966244 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:08.966336 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:08.966823 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:08.967375 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:08.967480 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:08.967609 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 02:51:08.967667 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":453,"threads_started":1}
I20260621 02:51:08.971338 19446 raft_consensus.cc:1261] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Refusing update from remote peer 7ed257eaeefc45a38cd4b2845db2a47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:08.971573 19587 raft_consensus.cc:1261] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Refusing update from remote peer 7ed257eaeefc45a38cd4b2845db2a47c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:08.972012 19877 consensus_queue.cc:1035] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.972374 19803 consensus_queue.cc:1035] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:08.977234 19841 mvcc.cc:205] Tried to move back new op lower bound from 7299114061699870720 to 7299114061597011968. Current Snapshot: MvccSnapshot[applied={T|T < 7299114061699870720}]
I20260621 02:51:08.992022 19284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:08.992326 19284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:08.994943 19284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Sending DeleteTablet for 3 replicas of tablet 50cd87393d5f449494bb0330a537f8da
I20260621 02:51:08.995599 19422 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cd87393d5f449494bb0330a537f8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60004
I20260621 02:51:08.995812 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cd87393d5f449494bb0330a537f8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:34486
I20260621 02:51:08.995903 19857 tablet_replica.cc:330] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: stopping tablet replica
I20260621 02:51:08.996004 19561 tablet_service.cc:1514] Processing DeleteTablet for tablet 50cd87393d5f449494bb0330a537f8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:08 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260621 02:51:08.996084 19856 tablet_replica.cc:330] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: stopping tablet replica
I20260621 02:51:08.996107 19857 raft_consensus.cc:2229] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:08.996263 19858 tablet_replica.cc:330] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: stopping tablet replica
I20260621 02:51:08.996263 19856 raft_consensus.cc:2229] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:51:08.996299 19857 raft_consensus.cc:2258] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.996438 19858 raft_consensus.cc:2229] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:08.996639 19858 raft_consensus.cc:2258] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.996711 19856 raft_consensus.cc:2258] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:08.997282 19857 ts_tablet_manager.cc:1905] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.998006 19856 ts_tablet_manager.cc:1905] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:08.998270 19284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:51:08.998888 19284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:09.000820 19858 ts_tablet_manager.cc:1905] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:09.004740 19857 ts_tablet_manager.cc:1918] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:09.004940 19857 log.cc:1196] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/50cd87393d5f449494bb0330a537f8da
I20260621 02:51:09.005407 19857 ts_tablet_manager.cc:1939] T 50cd87393d5f449494bb0330a537f8da P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting consensus metadata
I20260621 02:51:09.006414 19272 catalog_manager.cc:4841] TS 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613): tablet 50cd87393d5f449494bb0330a537f8da (table test-table [id=767ce0f84a464238a7d12c25776bbfd8]) successfully deleted
I20260621 02:51:09.006716 19856 ts_tablet_manager.cc:1918] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:09.006889 19856 log.cc:1196] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/50cd87393d5f449494bb0330a537f8da
I20260621 02:51:09.007295 19856 ts_tablet_manager.cc:1939] T 50cd87393d5f449494bb0330a537f8da P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting consensus metadata
I20260621 02:51:09.008039 19422 tablet_service.cc:1467] Processing CreateTablet for tablet 319e9f69a02940eb85d983a1b48cc063 (DEFAULT_TABLE table=test-table [id=fdf486ff120741ad977f2b098d00a1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:09.008438 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319e9f69a02940eb85d983a1b48cc063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:09.008361 19561 tablet_service.cc:1467] Processing CreateTablet for tablet 319e9f69a02940eb85d983a1b48cc063 (DEFAULT_TABLE table=test-table [id=fdf486ff120741ad977f2b098d00a1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:09.008694 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319e9f69a02940eb85d983a1b48cc063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:09.014021 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 319e9f69a02940eb85d983a1b48cc063 (DEFAULT_TABLE table=test-table [id=fdf486ff120741ad977f2b098d00a1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:09.014456 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 319e9f69a02940eb85d983a1b48cc063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:09.014762 19797 tablet_bootstrap.cc:492] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Bootstrap starting.
I20260621 02:51:09.015914 19797 tablet_bootstrap.cc:654] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:09.017860 19797 tablet_bootstrap.cc:492] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: No bootstrap required, opened a new log
I20260621 02:51:09.018043 19797 ts_tablet_manager.cc:1397] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:51:09.018620 19797 raft_consensus.cc:348] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.018841 19797 raft_consensus.cc:374] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:09.018920 19797 raft_consensus.cc:729] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ab78a2e85943b9820cdbf29ec3f4b3, State: Initialized, Role: FOLLOWER
I20260621 02:51:09.019170 19797 consensus_queue.cc:260] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.019562 19272 catalog_manager.cc:4841] TS 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067): tablet 50cd87393d5f449494bb0330a537f8da (table test-table [id=767ce0f84a464238a7d12c25776bbfd8]) successfully deleted
I20260621 02:51:09.020387 19797 ts_tablet_manager.cc:1428] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 02:51:09.021590 19858 ts_tablet_manager.cc:1918] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:09.021942 19796 tablet_bootstrap.cc:492] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Bootstrap starting.
I20260621 02:51:09.022147 19798 tablet_bootstrap.cc:492] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Bootstrap starting.
I20260621 02:51:09.023591 19796 tablet_bootstrap.cc:654] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:09.023830 19798 tablet_bootstrap.cc:654] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:09.026649 19798 tablet_bootstrap.cc:492] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: No bootstrap required, opened a new log
I20260621 02:51:09.026806 19798 ts_tablet_manager.cc:1397] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260621 02:51:09.026995 19858 log.cc:1196] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/50cd87393d5f449494bb0330a537f8da
I20260621 02:51:09.027376 19858 ts_tablet_manager.cc:1939] T 50cd87393d5f449494bb0330a537f8da P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting consensus metadata
I20260621 02:51:09.027575 19798 raft_consensus.cc:348] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.027833 19798 raft_consensus.cc:374] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:09.027915 19798 raft_consensus.cc:729] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed257eaeefc45a38cd4b2845db2a47c, State: Initialized, Role: FOLLOWER
I20260621 02:51:09.028173 19798 consensus_queue.cc:260] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.028429 19273 catalog_manager.cc:4841] TS 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457): tablet 50cd87393d5f449494bb0330a537f8da (table test-table [id=767ce0f84a464238a7d12c25776bbfd8]) successfully deleted
I20260621 02:51:09.028891 19798 ts_tablet_manager.cc:1428] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 02:51:09.034341 19796 tablet_bootstrap.cc:492] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: No bootstrap required, opened a new log
I20260621 02:51:09.034504 19796 ts_tablet_manager.cc:1397] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260621 02:51:09.035109 19796 raft_consensus.cc:348] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.035293 19796 raft_consensus.cc:374] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:09.035390 19796 raft_consensus.cc:729] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d59d2e132704b7bbe3c5dc5c1e2e117, State: Initialized, Role: FOLLOWER
I20260621 02:51:09.035627 19796 consensus_queue.cc:260] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.036152 19796 ts_tablet_manager.cc:1428] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 02:51:09.221673 19804 raft_consensus.cc:482] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:09.221860 19804 raft_consensus.cc:504] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.223409 19804 leader_election.cc:288] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:09.230675 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.224544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:38435 (local address 127.11.109.2:39457)
0621 02:51:09.224719 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.224725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.224742 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:51:09.224817 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.224840 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.224923 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.225021 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:09.225937 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.226875 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.228676 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.229691 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.229846 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.229919 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.230261 (+ 342us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.230405 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:51:09.230793 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.223565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39457 (local address 127.11.109.1:38435)
0621 02:51:09.224146 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:09.224171 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:09.224327 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:09.225235 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:09.225240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:09.225273 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:09.225744 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.225759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.226988 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.226997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.228530 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.228541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.229909 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.229916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.230094 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.230128 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:51:09.230233 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:51:09.230400 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260621 02:51:09.231572 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.223749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:44067 (local address 127.11.109.1:53585)
0621 02:51:09.224511 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:09.224532 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:09.224688 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:09.225036 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:09.225040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:09.225066 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:09.225421 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.225434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.226845 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.226852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.229023 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.229038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.230853 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.230858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.231013 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.231047 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:51:09.231140 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:51:09.231266 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":715,"threads_started":1}
I20260621 02:51:09.231568 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.224232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:53585 (local address 127.11.109.3:44067)
0621 02:51:09.224403 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.224410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.224599 (+ 189us) server_negotiation.cc:407] Connection header received
0621 02:51:09.224856 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.224860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.224920 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.225015 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:09.225580 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.226692 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.229168 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.230479 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.230612 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.230688 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.231181 (+ 493us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.231307 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:51:09.231563 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319e9f69a02940eb85d983a1b48cc063" candidate_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" is_pre_election: true
I20260621 02:51:09.231868 19587 raft_consensus.cc:2454] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d59d2e132704b7bbe3c5dc5c1e2e117 in term 0.
I20260621 02:51:09.232285 19720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319e9f69a02940eb85d983a1b48cc063" candidate_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" is_pre_election: true
I20260621 02:51:09.232322 19376 leader_election.cc:302] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:09.232486 19720 raft_consensus.cc:2454] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d59d2e132704b7bbe3c5dc5c1e2e117 in term 0.
I20260621 02:51:09.232741 19804 raft_consensus.cc:2781] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:09.232851 19804 raft_consensus.cc:482] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:09.232957 19804 raft_consensus.cc:3037] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:09.235154 19804 raft_consensus.cc:504] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.235716 19804 leader_election.cc:288] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [CANDIDATE]: Term 1 election: Requested vote from peers 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457), 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067)
I20260621 02:51:09.236137 19587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319e9f69a02940eb85d983a1b48cc063" candidate_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3"
I20260621 02:51:09.236212 19720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "319e9f69a02940eb85d983a1b48cc063" candidate_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed257eaeefc45a38cd4b2845db2a47c"
I20260621 02:51:09.236373 19587 raft_consensus.cc:3037] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:09.236466 19720 raft_consensus.cc:3037] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:09.238454 19587 raft_consensus.cc:2454] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d59d2e132704b7bbe3c5dc5c1e2e117 in term 1.
I20260621 02:51:09.238807 19376 leader_election.cc:302] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d59d2e132704b7bbe3c5dc5c1e2e117, 37ab78a2e85943b9820cdbf29ec3f4b3; no voters:
I20260621 02:51:09.239077 19804 raft_consensus.cc:2781] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:09.239563 19804 raft_consensus.cc:686] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 LEADER]: Becoming Leader. State: Replica: 0d59d2e132704b7bbe3c5dc5c1e2e117, State: Running, Role: LEADER
I20260621 02:51:09.239692 19720 raft_consensus.cc:2454] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d59d2e132704b7bbe3c5dc5c1e2e117 in term 1.
I20260621 02:51:09.240242 19804 consensus_queue.cc:237] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } }
I20260621 02:51:09.243574 19284 catalog_manager.cc:5495] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1). New cstate: current_term: 1 leader_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d59d2e132704b7bbe3c5dc5c1e2e117" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 37613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 } health_report { overall_health: UNKNOWN } } }
I20260621 02:51:09.273164 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.267502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.11.109.1:37613)
0621 02:51:09.267665 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.267669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.267684 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:51:09.267921 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.267924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.267992 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.268108 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:09.268557 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.269157 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.271059 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.271442 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.271534 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.272063 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:09.272578 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:09.272695 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.272840 (+ 145us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.272900 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:51:09.288635 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.282754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:56081 (local address 127.11.109.2:39457)
0621 02:51:09.282973 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.282981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.283491 (+ 510us) server_negotiation.cc:407] Connection header received
0621 02:51:09.283667 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.283671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.283757 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.283826 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:09.284402 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.285082 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.287059 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.287506 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.287623 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.288102 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:09.288190 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:09.288252 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.288402 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.288458 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:51:09.288851 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.282703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39457 (local address 127.11.109.1:56081)
0621 02:51:09.283387 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:09.283407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:09.283550 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:09.283874 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:09.283882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:09.283897 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:51:09.284265 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.284280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.285246 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.285252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.286928 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.286942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.287693 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.287699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.287863 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.287943 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:51:09.288287 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:51:09.288298 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:51:09.288388 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 02:51:09.288507 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":322,"threads_started":1}
I20260621 02:51:09.297621 19720 raft_consensus.cc:1261] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Refusing update from remote peer 0d59d2e132704b7bbe3c5dc5c1e2e117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:09.298136 19587 raft_consensus.cc:1261] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Refusing update from remote peer 0d59d2e132704b7bbe3c5dc5c1e2e117: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:09.298300 19890 consensus_queue.cc:1035] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed257eaeefc45a38cd4b2845db2a47c" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:09.298709 19804 consensus_queue.cc:1035] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ab78a2e85943b9820cdbf29ec3f4b3" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:09.332186 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.326760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:52639 (local address 127.11.109.1:37613)
0621 02:51:09.326920 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.326927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.326948 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:51:09.327005 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.327010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.327100 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.327210 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:09.327700 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.328820 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.330332 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.330943 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.331083 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.331443 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:09.331554 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:09.331649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.331794 (+ 145us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.331867 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:51:09.332247 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.325976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:37613 (local address 127.11.109.2:52639)
0621 02:51:09.326551 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:09.326570 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:09.326701 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:09.327237 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:09.327241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:09.327253 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:51:09.327541 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.327553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.328951 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.328955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.330198 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:09.330210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.331091 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.331096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:09.331258 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.331292 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:51:09.331649 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:51:09.331662 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 02:51:09.331769 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 02:51:09.331898 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":1346,"threads_started":1}
I20260621 02:51:09.364714 19284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:09.364996 19284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:09.368662 19284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9be0b377e87746fca9c7ac4bdf325a5e: Sending DeleteTablet for 3 replicas of tablet 319e9f69a02940eb85d983a1b48cc063
I20260621 02:51:09.369513 19561 tablet_service.cc:1514] Processing DeleteTablet for tablet 319e9f69a02940eb85d983a1b48cc063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:09 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260621 02:51:09.369534 19422 tablet_service.cc:1514] Processing DeleteTablet for tablet 319e9f69a02940eb85d983a1b48cc063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:09 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:60004
I20260621 02:51:09.369817 19858 tablet_replica.cc:330] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: stopping tablet replica
I20260621 02:51:09.369867 19857 tablet_replica.cc:330] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: stopping tablet replica
I20260621 02:51:09.370011 19858 raft_consensus.cc:2229] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:09.370065 19857 raft_consensus.cc:2229] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:51:09.370213 19858 raft_consensus.cc:2258] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:09.370597 19857 raft_consensus.cc:2258] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:09.371524 19858 ts_tablet_manager.cc:1905] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:09.371496 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 319e9f69a02940eb85d983a1b48cc063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:09 UTC) from {username='oryx', principal='oryx/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:34486
I20260621 02:51:09.371824 19856 tablet_replica.cc:330] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: stopping tablet replica
I20260621 02:51:09.372004 19856 raft_consensus.cc:2229] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:09.372247 19856 raft_consensus.cc:2258] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:09.375155 19856 ts_tablet_manager.cc:1905] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:09.377523 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 19357
I20260621 02:51:09.380023 19857 ts_tablet_manager.cc:1905] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:09.383623 19858 ts_tablet_manager.cc:1918] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:51:09.383785 19858 log.cc:1196] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/319e9f69a02940eb85d983a1b48cc063
I20260621 02:51:09.384219 19858 ts_tablet_manager.cc:1939] T 319e9f69a02940eb85d983a1b48cc063 P 37ab78a2e85943b9820cdbf29ec3f4b3: Deleting consensus metadata
I20260621 02:51:09.385330 19273 catalog_manager.cc:4841] TS 37ab78a2e85943b9820cdbf29ec3f4b3 (127.11.109.2:39457): tablet 319e9f69a02940eb85d983a1b48cc063 (table test-table [id=fdf486ff120741ad977f2b098d00a1cc]) successfully deleted
I20260621 02:51:09.388880 19856 ts_tablet_manager.cc:1918] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:51:09.389006 19856 log.cc:1196] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/319e9f69a02940eb85d983a1b48cc063
I20260621 02:51:09.392760 19857 ts_tablet_manager.cc:1918] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:51:09.392935 19857 log.cc:1196] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/319e9f69a02940eb85d983a1b48cc063
I20260621 02:51:09.393662 19857 ts_tablet_manager.cc:1939] T 319e9f69a02940eb85d983a1b48cc063 P 0d59d2e132704b7bbe3c5dc5c1e2e117: Deleting consensus metadata
I20260621 02:51:09.393913 19856 ts_tablet_manager.cc:1939] T 319e9f69a02940eb85d983a1b48cc063 P 7ed257eaeefc45a38cd4b2845db2a47c: Deleting consensus metadata
I20260621 02:51:09.395941 19272 catalog_manager.cc:4841] TS 7ed257eaeefc45a38cd4b2845db2a47c (127.11.109.3:44067): tablet 319e9f69a02940eb85d983a1b48cc063 (table test-table [id=fdf486ff120741ad977f2b098d00a1cc]) successfully deleted
I20260621 02:51:09.397243 19272 catalog_manager.cc:4841] TS 0d59d2e132704b7bbe3c5dc5c1e2e117 (127.11.109.1:37613): tablet 319e9f69a02940eb85d983a1b48cc063 (table test-table [id=fdf486ff120741ad977f2b098d00a1cc]) successfully deleted
Jun 21 02:51:09 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.1@KRBTEST.COM
I20260621 02:51:09.409593 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.378755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.11.109.1:37613)
0621 02:51:09.379193 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.379199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.390539 (+ 11340us) server_negotiation.cc:407] Connection header received
0621 02:51:09.390984 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.390990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.391075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.391166 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:09.392986 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.394202 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.396109 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.396805 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.396902 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.402233 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:09.402246 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:09.402264 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:09.402340 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:09.406488 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:09.407271 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:09.407282 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:09.407286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:09.407465 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:09.407876 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:09.407882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:09.407885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:09.408366 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:09.408684 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.408954 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.409228 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260621 02:51:09.410347 19488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:09.517880 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 19357
I20260621 02:51:09.537439 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 19497
Jun 21 02:51:09 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.2@KRBTEST.COM
I20260621 02:51:09.551816 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.538347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.11.109.2:39457)
0621 02:51:09.538546 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.538554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.538574 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:51:09.538623 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.538626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.538687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.538792 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:09.540023 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.540683 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.542090 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.542496 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.542628 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.546531 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:09.546543 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:09.546558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:09.546589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:09.549433 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:09.550036 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:09.550043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:09.550045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:09.550176 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:09.550537 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:09.550540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:09.550542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:09.550848 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:09.551062 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.551293 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.551464 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:51:09.552443 19627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:09.692862 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 19497
I20260621 02:51:09.718472 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 19636
Jun 21 02:51:09 dist-test-slave-bc52 krb5kdc[19241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.109.3@KRBTEST.COM
I20260621 02:51:09.733119 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:09.719160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.11.109.3:44067)
0621 02:51:09.719383 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:09.719390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:09.719478 (+ 88us) server_negotiation.cc:407] Connection header received
0621 02:51:09.719691 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:09.719696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:09.719769 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:09.719873 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:09.721129 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.721896 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.723170 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:09.723611 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:09.723761 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:09.728011 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:09.728025 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:09.728042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:09.728077 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:09.730923 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:09.731573 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:09.731580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:09.731582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:09.731714 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:09.732008 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:09.732011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:09.732013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:09.732323 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:09.732533 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:09.732736 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 02:51:09.732884 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:51:09.733773 19766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:09.857079 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 19636
I20260621 02:51:09.878145 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 19254
I20260621 02:51:09.879305 19326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:09.993597 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 19254
2026-06-21T02:51:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6742 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:51:10 dist-test-slave-bc52 krb5kdc[19911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:51:10 dist-test-slave-bc52 krb5kdc[19911](info): set up 2 sockets
Jun 21 02:51:10 dist-test-slave-bc52 krb5kdc[19911](info): commencing operation
krb5kdc: starting...
W20260621 02:51:11.074151 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:51:11 dist-test-slave-bc52 krb5kdc[19911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:51:11.272823 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37959
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:37959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:11.585021 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:11.585356 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:11.585533 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:11.592525 19924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:11.592610 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:11.592705 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:11.592794 19924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:11.592872 19924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:11.602216 19924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:37959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:11.603369 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:11.604651 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:11.616432 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:11.616451 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:11.616590 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:11.617542 19924 server_base.cc:1034] running on GCE node
I20260621 02:51:11.618266 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:11.619632 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:11.620822 19924 hybrid_clock.cc:648] HybridClock initialized: now 1782010271620788 us; error 24 us; skew 500 ppm
I20260621 02:51:11.621328 19924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T02:51:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1697 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:51:11 dist-test-slave-bc52 krb5kdc[19938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:51:11 dist-test-slave-bc52 krb5kdc[19938](info): set up 2 sockets
Jun 21 02:51:11 dist-test-slave-bc52 krb5kdc[19938](info): commencing operation
krb5kdc: starting...
W20260621 02:51:12.773994 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:51:12 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:51:12.978495 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45409
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:13.292618 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:13.292932 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:13.293085 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:13.300391 19951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:13.300556 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:13.300679 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:13.300776 19951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:13.300865 19951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:13.310387 19951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:13.311885 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:13.313272 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:13.324843 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:13.324806 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:13.324995 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:13.325899 19951 server_base.cc:1034] running on GCE node
I20260621 02:51:13.326709 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:13.327986 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:13.329214 19951 hybrid_clock.cc:648] HybridClock initialized: now 1782010273329131 us; error 38 us; skew 500 ppm
I20260621 02:51:13.329525 19951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:13 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010273, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:13.336007 19951 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:51:13.337528 19951 webserver.cc:460] Webserver started at http://127.11.109.62:44577/ using document root <none> and password file <none>
I20260621 02:51:13.338234 19951 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:13.338387 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:13.338706 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:13.341843 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "5249fe2f44fd4fa8b8b322b606f50d3e"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "b9945dadcdece23285dac99184a33822"
server_key_iv: "c5c42cb21b21bc53cad3d849c0637aa3"
server_key_version: "encryptionkey@0"
I20260621 02:51:13.342658 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "5249fe2f44fd4fa8b8b322b606f50d3e"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "b9945dadcdece23285dac99184a33822"
server_key_iv: "c5c42cb21b21bc53cad3d849c0637aa3"
server_key_version: "encryptionkey@0"
I20260621 02:51:13.348071 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:51:13.351507 19966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:13.352778 19951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:51:13.352970 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "5249fe2f44fd4fa8b8b322b606f50d3e"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "b9945dadcdece23285dac99184a33822"
server_key_iv: "c5c42cb21b21bc53cad3d849c0637aa3"
server_key_version: "encryptionkey@0"
I20260621 02:51:13.353165 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:13.378983 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:13.383033 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:13.383309 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:13.401517 19951 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45409
I20260621 02:51:13.401537 20017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45409 every 8 connection(s)
I20260621 02:51:13.402813 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:51:13.406293 20018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:13.408362 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 19951
I20260621 02:51:13.408596 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:51:13.409281 11700 external_mini_cluster.cc:1428] Setting key 93be7787e7c6c818aff0e3bbae891208
I20260621 02:51:13.417539 20018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Bootstrap starting.
I20260621 02:51:13.420464 20018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:13.421489 20018 log.cc:824] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:51:13 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:13.424510 20018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: No bootstrap required, opened a new log
I20260621 02:51:13.430799 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:13.411312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.11.109.62:45409)
0621 02:51:13.412160 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:13.412173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:13.412212 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:51:13.413930 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:13.413952 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:13.414318 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:13.414726 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:13.416178 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:13.417499 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:13.418699 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:13.419403 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:13.419589 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:13.423829 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:13.423835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:13.423848 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:13.423880 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:13.426879 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:13.427655 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:13.427660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:13.427666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:13.427825 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:13.428240 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:13.428246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:13.428248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:13.428690 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:13.428901 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:13.429427 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 02:51:13.429806 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":431,"threads_started":1}
I20260621 02:51:13.434130 20018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:13.434572 20018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:13.434679 20018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5249fe2f44fd4fa8b8b322b606f50d3e, State: Initialized, Role: FOLLOWER
I20260621 02:51:13.435266 20018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:13.435472 20018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:51:13.435565 20018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:51:13.435681 20018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:13.437772 20018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:13.438185 20018 leader_election.cc:302] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5249fe2f44fd4fa8b8b322b606f50d3e; no voters:
I20260621 02:51:13.438760 20018 leader_election.cc:288] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:51:13.438900 20023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:13.440237 20023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 LEADER]: Becoming Leader. State: Replica: 5249fe2f44fd4fa8b8b322b606f50d3e, State: Running, Role: LEADER
I20260621 02:51:13.440799 20023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:13.440920 20018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:51:13.443859 20025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5249fe2f44fd4fa8b8b322b606f50d3e. Latest consensus state: current_term: 1 leader_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } } }
I20260621 02:51:13.444240 20025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:13.444789 20024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } } }
I20260621 02:51:13.445119 20024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:13.451313 20032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:13.456205 20032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:13.465482 20032 catalog_manager.cc:1347] Generated new cluster ID: eaab3dd264ce4e1f81a696f1a6009b37
I20260621 02:51:13.465595 20032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:13.492492 20032 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:51:13.493892 20032 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:13.501396 20032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Generated new TSK 0
I20260621 02:51:13.502187 20032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:51:13.561477 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45409
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:13.882210 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:13.882483 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:13.882571 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:13.889489 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:13.889581 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:13.889742 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:51:13.898902 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:13.900084 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:13.901486 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:13.914232 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:13.914206 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:13.914645 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:13.914960 20044 server_base.cc:1034] running on GCE node
I20260621 02:51:13.915509 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:13.916347 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:13.917578 20044 hybrid_clock.cc:648] HybridClock initialized: now 1782010273917545 us; error 59 us; skew 500 ppm
I20260621 02:51:13.917909 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:13 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010273, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:13.920845 20044 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:51:13.922224 20044 webserver.cc:460] Webserver started at http://127.11.109.1:43075/ using document root <none> and password file <none>
I20260621 02:51:13.922844 20044 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:13.922935 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:13.923171 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:13.925761 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "d17592219f8f45cd81a97a02e19bb8fd"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "1b162116524c4700d6e78bb9144fb69e"
server_key_iv: "82b08c9d30ab7725180c46796c53ed2b"
server_key_version: "encryptionkey@0"
I20260621 02:51:13.926451 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "d17592219f8f45cd81a97a02e19bb8fd"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "1b162116524c4700d6e78bb9144fb69e"
server_key_iv: "82b08c9d30ab7725180c46796c53ed2b"
server_key_version: "encryptionkey@0"
I20260621 02:51:13.931249 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:51:13.934502 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:13.935853 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:51:13.936010 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "d17592219f8f45cd81a97a02e19bb8fd"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "1b162116524c4700d6e78bb9144fb69e"
server_key_iv: "82b08c9d30ab7725180c46796c53ed2b"
server_key_version: "encryptionkey@0"
I20260621 02:51:13.936203 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:13.983369 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:13.987452 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:13.987722 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:13.988619 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:13.989984 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:13.990056 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:13.990191 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:13.990240 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.023144 20044 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:42783
I20260621 02:51:14.023226 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:42783 every 8 connection(s)
I20260621 02:51:14.024425 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:51:14.033296 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20044
I20260621 02:51:14.033494 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:51:14.034248 11700 external_mini_cluster.cc:1428] Setting key 313c0b3c78666d2afccda1933e659cb4
Jun 21 02:51:14 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010273, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:14.045661 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:14.026906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:49653 (local address 127.11.109.62:45409)
0621 02:51:14.027358 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:14.027364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:14.028253 (+ 889us) server_negotiation.cc:407] Connection header received
0621 02:51:14.030025 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:14.030030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:14.030112 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:14.030206 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:14.031570 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.032280 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.034428 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.034861 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.035028 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:14.038755 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:14.038763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:14.038770 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:14.038813 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:14.042401 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:14.043145 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:14.043153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:14.043159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:14.043316 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:14.043801 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:14.043805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:14.043807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:14.044095 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:14.044200 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:14.045058 (+ 858us) server_negotiation.cc:299] Negotiation successful
0621 02:51:14.045292 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260621 02:51:14.046528 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:14.027447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:49653)
0621 02:51:14.028103 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:14.028144 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:14.029784 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:14.030412 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:14.030420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:14.030881 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:14.031423 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:14.031439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.032410 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.032413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:14.034285 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:14.034312 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.035126 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.035133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:14.035384 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:14.036069 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:14.036103 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:14.038559 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:14.042542 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:14.042552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:14.042572 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:14.043015 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:14.043442 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:14.043449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:14.043455 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:14.043688 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:14.044217 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:14.044227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:14.044640 (+ 413us) client_negotiation.cc:769] Sending connection context
0621 02:51:14.044949 (+ 309us) client_negotiation.cc:240] Negotiation successful
0621 02:51:14.045468 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":266,"threads_started":1}
I20260621 02:51:14.048022 20172 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45409
I20260621 02:51:14.048295 20172 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:14.048961 20172 heartbeater.cc:511] Master 127.11.109.62:45409 requested a full tablet report, sending...
I20260621 02:51:14.051260 19983 ts_manager.cc:194] Registered new tserver with Master: d17592219f8f45cd81a97a02e19bb8fd (127.11.109.1:42783)
I20260621 02:51:14.053478 19983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:49653
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:51:14.083685 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45409
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:14.391994 20178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:14.392244 20178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:14.392369 20178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:14.399510 20178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:14.399607 20178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:14.399784 20178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:51:14.409488 20178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:14.410760 20178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:14.412009 20178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:14.424775 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:14.424691 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:14.424726 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:14.425530 20178 server_base.cc:1034] running on GCE node
I20260621 02:51:14.426118 20178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:14.427016 20178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:14.428236 20178 hybrid_clock.cc:648] HybridClock initialized: now 1782010274428185 us; error 59 us; skew 500 ppm
I20260621 02:51:14.428550 20178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:14 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010274, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:14.431586 20178 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:51:14.432999 20178 webserver.cc:460] Webserver started at http://127.11.109.2:35407/ using document root <none> and password file <none>
I20260621 02:51:14.433807 20178 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:14.433945 20178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:14.434231 20178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:14.437669 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "af5d22ef989a4c17b7320e0fac1b8423"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "c8f066683100b97185da152e3ba116ed"
server_key_iv: "fd45c77551b0d2ae8994ea12774b6f45"
server_key_version: "encryptionkey@0"
I20260621 02:51:14.438656 20178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "af5d22ef989a4c17b7320e0fac1b8423"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "c8f066683100b97185da152e3ba116ed"
server_key_iv: "fd45c77551b0d2ae8994ea12774b6f45"
server_key_version: "encryptionkey@0"
I20260621 02:51:14.443612 20178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:51:14.446923 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.448173 20178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:51:14.448311 20178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "af5d22ef989a4c17b7320e0fac1b8423"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "c8f066683100b97185da152e3ba116ed"
server_key_iv: "fd45c77551b0d2ae8994ea12774b6f45"
server_key_version: "encryptionkey@0"
I20260621 02:51:14.448467 20178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:14.467329 20178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:14.470963 20178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:14.471181 20178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:14.471995 20178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:14.473165 20178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:14.473258 20178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.473351 20178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:14.473397 20178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.505682 20178 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:42557
I20260621 02:51:14.505725 20305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:42557 every 8 connection(s)
I20260621 02:51:14.506908 20178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:51:14.512380 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20178
I20260621 02:51:14.512568 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:51:14.513017 11700 external_mini_cluster.cc:1428] Setting key e2da4c421b2a935baff03f04118b3cc7
Jun 21 02:51:14 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010274, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:14.527601 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:14.509434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:39663 (local address 127.11.109.62:45409)
0621 02:51:14.509609 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:14.509614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:14.510794 (+ 1180us) server_negotiation.cc:407] Connection header received
0621 02:51:14.512567 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:14.512573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:14.512695 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:14.512816 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:14.514570 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.515575 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.517087 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.517822 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.517960 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:14.521550 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:14.521558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:14.521566 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:14.521611 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:14.524197 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:14.525124 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:14.525128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:14.525130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:14.525357 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:14.525842 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:14.525845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:14.525847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:14.526179 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:14.526311 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:14.527161 (+ 850us) server_negotiation.cc:299] Negotiation successful
0621 02:51:14.527350 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:51:14.529312 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:14.509948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:39663)
0621 02:51:14.510603 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:14.510640 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:14.512286 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:14.513069 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:14.513077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:14.513635 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:14.514411 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:14.514435 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.515725 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.515729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:14.516942 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:14.516954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.518042 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.518048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:14.518242 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:14.518944 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:14.518970 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:14.521347 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:14.524330 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:14.524341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:14.524361 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:14.524989 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:14.525465 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:14.525472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:14.525478 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:14.525707 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:14.526310 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:14.526321 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:14.526770 (+ 449us) client_negotiation.cc:769] Sending connection context
0621 02:51:14.527062 (+ 292us) client_negotiation.cc:240] Negotiation successful
0621 02:51:14.527660 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":253,"threads_started":1}
I20260621 02:51:14.530963 20306 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45409
I20260621 02:51:14.531296 20306 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:14.532037 20306 heartbeater.cc:511] Master 127.11.109.62:45409 requested a full tablet report, sending...
I20260621 02:51:14.533552 19983 ts_manager.cc:194] Registered new tserver with Master: af5d22ef989a4c17b7320e0fac1b8423 (127.11.109.2:42557)
I20260621 02:51:14.534451 19983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:39663
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:51:14.565493 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45409
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:14.875123 20311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:14.875451 20311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:14.875623 20311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:14.882649 20311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:14.882808 20311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:14.883064 20311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:51:14.892378 20311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:14.893667 20311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:14.894917 20311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:14.907531 20316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:14.907593 20319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:14.907618 20317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:14.908334 20311 server_base.cc:1034] running on GCE node
I20260621 02:51:14.908947 20311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:14.909888 20311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:14.911123 20311 hybrid_clock.cc:648] HybridClock initialized: now 1782010274911083 us; error 53 us; skew 500 ppm
I20260621 02:51:14.911427 20311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:14 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010274, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:14.914445 20311 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:51:14.915810 20311 webserver.cc:460] Webserver started at http://127.11.109.3:36833/ using document root <none> and password file <none>
I20260621 02:51:14.916420 20311 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:14.916520 20311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:14.916756 20311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:14.919440 20311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "92a60b709705406f9d0d3af1404e251e"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "fc2a2c869f82b7a07701fc42b570a802"
server_key_iv: "34ef07415db821045337c7042fbd4899"
server_key_version: "encryptionkey@0"
I20260621 02:51:14.920140 20311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "92a60b709705406f9d0d3af1404e251e"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "fc2a2c869f82b7a07701fc42b570a802"
server_key_iv: "34ef07415db821045337c7042fbd4899"
server_key_version: "encryptionkey@0"
I20260621 02:51:14.925052 20311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260621 02:51:14.928231 20326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.929605 20311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:51:14.929772 20311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "92a60b709705406f9d0d3af1404e251e"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "fc2a2c869f82b7a07701fc42b570a802"
server_key_iv: "34ef07415db821045337c7042fbd4899"
server_key_version: "encryptionkey@0"
I20260621 02:51:14.929970 20311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:14.949958 20311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:14.953982 20311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:14.954249 20311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:14.955123 20311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:14.956480 20311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:14.956566 20311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.956737 20311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:14.956785 20311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:14.988059 20311 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:40943
I20260621 02:51:14.988131 20438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:40943 every 8 connection(s)
I20260621 02:51:14.989351 20311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:51:14.994884 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20311
I20260621 02:51:14.995103 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:51:14.995632 11700 external_mini_cluster.cc:1428] Setting key d60006acb5a89d8a5d2bd6689f5a8228
Jun 21 02:51:15 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010274, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:15.009213 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:14.991659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:53747 (local address 127.11.109.62:45409)
0621 02:51:14.991854 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:14.991858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:14.992980 (+ 1122us) server_negotiation.cc:407] Connection header received
0621 02:51:14.994877 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:14.994882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:14.994975 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:14.995106 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:14.996991 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.998090 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.999759 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.000396 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.000533 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:15.003955 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:15.003959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:15.003963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:15.003995 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:15.006475 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.007142 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.007146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.007148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.007280 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.007641 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.007644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.007646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.007903 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:15.008023 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:15.008644 (+ 621us) server_negotiation.cc:299] Negotiation successful
0621 02:51:15.008796 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:51:15.010131 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:14.992116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:53747)
0621 02:51:14.992803 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:14.992845 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:14.994569 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:14.995369 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:14.995381 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:14.996080 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:14.996817 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:14.996843 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:14.998224 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:14.998228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:14.999632 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:14.999644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.000641 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.000648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:15.001002 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:15.001695 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:15.001717 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:15.003774 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:15.006592 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:15.006598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:15.006609 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:15.007004 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:15.007381 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:15.007384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:15.007386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:15.007551 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:15.008005 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:15.008011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:15.008286 (+ 275us) client_negotiation.cc:769] Sending connection context
0621 02:51:15.008504 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 02:51:15.008875 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260621 02:51:15.011408 20439 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45409
I20260621 02:51:15.011672 20439 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:15.012286 20439 heartbeater.cc:511] Master 127.11.109.62:45409 requested a full tablet report, sending...
I20260621 02:51:15.013638 19983 ts_manager.cc:194] Registered new tserver with Master: 92a60b709705406f9d0d3af1404e251e (127.11.109.3:40943)
I20260621 02:51:15.014487 19983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:53747
I20260621 02:51:15.023375 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.1@KRBTEST.COM" created.
I20260621 02:51:15.056373 20172 heartbeater.cc:503] Master 127.11.109.62:45409 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab.
Entry for principal oryx/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab.
WARNING: no policy specified for oryx/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab.
Entry for principal oryx/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab.
WARNING: no policy specified for oryx/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab.
Entry for principal oryx/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab.
I20260621 02:51:15.152407 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20044
Jun 21 02:51:15 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:51:15.168577 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:15.153687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.11.109.1:42783)
0621 02:51:15.154352 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:15.154360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:15.154385 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:51:15.154443 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:15.154450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:15.154666 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:15.154835 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:15.156241 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.157345 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.158613 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.159215 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.159353 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:15.163328 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:15.163338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:15.163353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:15.163388 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:15.166247 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.166833 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.166839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.166841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.166956 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.167253 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.167255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.167257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.167562 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:15.167800 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:15.168005 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 02:51:15.168184 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":287,"threads_started":1}
I20260621 02:51:15.169824 20167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:15.250607 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20044
I20260621 02:51:15.266925 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20178
Jun 21 02:51:15 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:51:15.282418 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:15.267988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.11.109.2:42557)
0621 02:51:15.268500 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:15.268507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:15.268531 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:51:15.268589 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:15.268595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:15.268830 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:15.268970 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:15.270173 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.271147 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.272294 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.272818 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.272909 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:15.276999 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:15.277009 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:15.277023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:15.277050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:15.279947 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.280530 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.280536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.280539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.280656 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.280947 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.280951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.280952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.281371 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:15.281597 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:15.281838 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 02:51:15.282019 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260621 02:51:15.283593 20301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:15.367547 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20178
I20260621 02:51:15.384742 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20311
Jun 21 02:51:15 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:51:15.400668 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:15.385807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.11.109.3:40943)
0621 02:51:15.386437 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:15.386443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:15.386472 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:51:15.386529 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:15.386535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:15.386748 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:15.386916 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:15.388296 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.389354 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.390620 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:15.391160 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:15.391301 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:15.395597 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:15.395608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:15.395621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:15.395656 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:15.398345 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.398989 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.398995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.398997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.399116 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:15.399463 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:15.399466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:15.399468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:15.399765 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:15.399996 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:15.400226 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 02:51:15.400403 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":333,"threads_started":1}
I20260621 02:51:15.402247 20434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:15.486377 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20311
I20260621 02:51:15.501801 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 19951
I20260621 02:51:15.502964 20013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:15.603223 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 19951
I20260621 02:51:15.623739 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45409
--webserver_interface=127.11.109.62
--webserver_port=44577
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:15.935984 20456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:15.936239 20456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:15.936384 20456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:15.943224 20456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:15.943315 20456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:15.943357 20456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:15.943431 20456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:15.943493 20456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:15.952908 20456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=44577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:15.954157 20456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:15.955382 20456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:15.967124 20461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:15.967044 20462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:15.967092 20464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:15.967733 20456 server_base.cc:1034] running on GCE node
I20260621 02:51:15.968472 20456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:15.969830 20456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:15.971026 20456 hybrid_clock.cc:648] HybridClock initialized: now 1782010275970965 us; error 50 us; skew 500 ppm
I20260621 02:51:15.971362 20456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:15 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:15.974558 20456 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:51:15.975903 20456 webserver.cc:460] Webserver started at http://127.11.109.62:44577/ using document root <none> and password file <none>
I20260621 02:51:15.976568 20456 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:15.976711 20456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:15.981945 20456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:51:15.984618 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:15.985891 20456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:51:15.986073 20456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "5249fe2f44fd4fa8b8b322b606f50d3e"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "b9945dadcdece23285dac99184a33822"
server_key_iv: "c5c42cb21b21bc53cad3d849c0637aa3"
server_key_version: "encryptionkey@0"
I20260621 02:51:15.987018 20456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:16.006888 20456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:16.011313 20456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:16.011591 20456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:16.029618 20456 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45409
I20260621 02:51:16.029636 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45409 every 8 connection(s)
I20260621 02:51:16.031073 20456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:51:16.033577 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20456
I20260621 02:51:16.034343 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:42783
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:45409
--webserver_port=43075
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:51:16.037894 20523 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:51:16.041132 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Bootstrap starting.
I20260621 02:51:16.068455 20523 log.cc:824] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:16.078917 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:51:16.079520 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Bootstrap complete.
I20260621 02:51:16.093309 20523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:16.094501 20523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5249fe2f44fd4fa8b8b322b606f50d3e, State: Initialized, Role: FOLLOWER
I20260621 02:51:16.095220 20523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:16.095489 20523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:51:16.095602 20523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:51:16.095775 20523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:51:16.102416 20523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:16.103050 20523 leader_election.cc:302] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5249fe2f44fd4fa8b8b322b606f50d3e; no voters:
I20260621 02:51:16.103662 20523 leader_election.cc:288] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:51:16.103811 20527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:51:16.105159 20527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [term 2 LEADER]: Becoming Leader. State: Replica: 5249fe2f44fd4fa8b8b322b606f50d3e, State: Running, Role: LEADER
I20260621 02:51:16.105687 20523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:51:16.105708 20527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } }
I20260621 02:51:16.108299 20528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } } }
I20260621 02:51:16.108242 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5249fe2f44fd4fa8b8b322b606f50d3e. Latest consensus state: current_term: 2 leader_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5249fe2f44fd4fa8b8b322b606f50d3e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45409 } } }
I20260621 02:51:16.108603 20528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:16.108670 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:16.121001 20533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:16.126883 20533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:16.128242 20533 catalog_manager.cc:1259] Loaded cluster ID: eaab3dd264ce4e1f81a696f1a6009b37
I20260621 02:51:16.128389 20533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:16.132251 20533 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:16.133658 20533 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5249fe2f44fd4fa8b8b322b606f50d3e: Loaded TSK: 0
I20260621 02:51:16.135167 20533 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:51:16.390409 20525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:16.390650 20525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:16.390784 20525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:16.397967 20525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:16.398061 20525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:16.398218 20525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:51:16.407833 20525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.1.keytab
--principal=oryx/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=43075
--tserver_master_addrs=127.11.109.62:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:16.409072 20525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:16.410384 20525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:16.423429 20549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:16.423486 20550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:16.424137 20552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:16.424479 20525 server_base.cc:1034] running on GCE node
I20260621 02:51:16.425071 20525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:16.426050 20525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:16.427263 20525 hybrid_clock.cc:648] HybridClock initialized: now 1782010276427215 us; error 40 us; skew 500 ppm
I20260621 02:51:16.427567 20525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:16 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010276, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:16.430745 20525 init.cc:377] Logged in from keytab as oryx/127.11.109.1@KRBTEST.COM (short username oryx)
I20260621 02:51:16.432231 20525 webserver.cc:460] Webserver started at http://127.11.109.1:43075/ using document root <none> and password file <none>
I20260621 02:51:16.432889 20525 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:16.433022 20525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:16.438203 20525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:51:16.440833 20559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:16.442453 20525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:51:16.442610 20525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "d17592219f8f45cd81a97a02e19bb8fd"
format_stamp: "Formatted at 2026-06-21 02:51:13 on dist-test-slave-bc52"
server_key: "1b162116524c4700d6e78bb9144fb69e"
server_key_iv: "82b08c9d30ab7725180c46796c53ed2b"
server_key_version: "encryptionkey@0"
I20260621 02:51:16.443992 20525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:16.460563 20525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:16.464771 20525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:16.464987 20525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:16.465952 20525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:16.467335 20525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:16.467429 20525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:16.467507 20525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:16.467567 20525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:16.499955 20525 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:42783
I20260621 02:51:16.500052 20671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:42783 every 8 connection(s)
I20260621 02:51:16.501473 20525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:51:16.507789 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20525
I20260621 02:51:16.508371 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:42557
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:45409
--webserver_port=35407
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:51:16 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:16 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:51:16.828696 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:16.828991 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:16.829169 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:16.836257 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:16.836400 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:16.836581 20676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:51:16.846601 20676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.2.keytab
--principal=oryx/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:42557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=35407
--tserver_master_addrs=127.11.109.62:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:16.847973 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:16.849426 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:16.866876 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:16.867035 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:16.867185 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:16.868645 20676 server_base.cc:1034] running on GCE node
I20260621 02:51:16.869769 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:16.871416 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:16.872716 20676 hybrid_clock.cc:648] HybridClock initialized: now 1782010276872619 us; error 61 us; skew 500 ppm
I20260621 02:51:16.873337 20676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:16 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010276, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:16.878324 20676 init.cc:377] Logged in from keytab as oryx/127.11.109.2@KRBTEST.COM (short username oryx)
I20260621 02:51:16.879938 20676 webserver.cc:460] Webserver started at http://127.11.109.2:35407/ using document root <none> and password file <none>
I20260621 02:51:16.880676 20676 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:16.880816 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:16.886637 20676 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:51:16.889882 20692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:16.891718 20676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260621 02:51:16.891918 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "af5d22ef989a4c17b7320e0fac1b8423"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "c8f066683100b97185da152e3ba116ed"
server_key_iv: "fd45c77551b0d2ae8994ea12774b6f45"
server_key_version: "encryptionkey@0"
I20260621 02:51:16.892894 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:16.912720 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:16.917126 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:16.917451 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:16.918551 20676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:16.920146 20676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:16.920291 20676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:16.920424 20676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:16.920512 20676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:16.966187 20676 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:42557
I20260621 02:51:16.966244 20804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:42557 every 8 connection(s)
I20260621 02:51:16.967641 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:51:16.968820 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20676
I20260621 02:51:16.969529 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:40943
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:45409
--webserver_port=36833
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:51:16 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:16 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:51:17.311343 20809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:17.311674 20809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:17.311859 20809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:17.318984 20809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:17.319108 20809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:17.319278 20809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:51:17.329414 20809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/krb5kdc/oryx_127.11.109.3.keytab
--principal=oryx/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:40943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=36833
--tserver_master_addrs=127.11.109.62:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:17.330718 20809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:17.331987 20809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:17.348349 20816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:17.348242 20815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:17.355023 20809 server_base.cc:1034] running on GCE node
W20260621 02:51:17.355249 20819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:17.355931 20809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:17.356948 20809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:17.358171 20809 hybrid_clock.cc:648] HybridClock initialized: now 1782010277358106 us; error 64 us; skew 500 ppm
I20260621 02:51:17.358518 20809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010277, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:17.361724 20809 init.cc:377] Logged in from keytab as oryx/127.11.109.3@KRBTEST.COM (short username oryx)
I20260621 02:51:17.365725 20809 webserver.cc:460] Webserver started at http://127.11.109.3:36833/ using document root <none> and password file <none>
I20260621 02:51:17.366436 20809 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:17.366590 20809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:17.372008 20809 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:51:17.374745 20825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:17.376179 20809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:51:17.376355 20809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "92a60b709705406f9d0d3af1404e251e"
format_stamp: "Formatted at 2026-06-21 02:51:14 on dist-test-slave-bc52"
server_key: "fc2a2c869f82b7a07701fc42b570a802"
server_key_iv: "34ef07415db821045337c7042fbd4899"
server_key_version: "encryptionkey@0"
I20260621 02:51:17.377302 20809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:17.404194 20809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:17.408591 20809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:17.408862 20809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:17.409955 20809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:17.411414 20809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:17.411540 20809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:17.411675 20809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:17.411763 20809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:17.458195 20809 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:40943
I20260621 02:51:17.458240 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:40943 every 8 connection(s)
I20260621 02:51:17.459617 20809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:51:17.461546 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 20809
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:17.520031 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:17.462677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.11.109.62:45409)
0621 02:51:17.477654 (+ 14977us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:17.477668 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:17.483511 (+ 5843us) server_negotiation.cc:407] Connection header received
0621 02:51:17.483733 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:17.483741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:17.483843 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:17.484003 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:17.486067 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.487998 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.490740 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.491665 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.491848 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.497083 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:17.497099 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:17.497127 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:17.497244 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:17.502840 (+ 5596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:17.514830 (+ 11990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:17.514845 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:17.514859 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:17.515216 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:17.515856 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:17.515864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:17.515870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:17.516723 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:17.517202 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:17.517940 (+ 738us) server_negotiation.cc:299] Negotiation successful
0621 02:51:17.518607 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14799,"thread_start_us":20653,"threads_started":1}
W20260621 02:51:17.840360 20675 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.326s user 0.005s sys 0.008s
W20260621 02:51:17.842300 20674 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:16.504329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:60695 (local address 127.11.109.62:45409)
0621 02:51:16.505118 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:16.505134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:16.505405 (+ 271us) server_negotiation.cc:407] Connection header received
0621 02:51:16.507529 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:16.507561 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:16.507942 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:16.508305 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:16.509834 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.511032 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.512557 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.513278 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.513477 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.841827 (+1328350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:60695: BlockingRecv error: recv got EOF from 127.11.109.1:60695 (error 108)
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":275,"threads_started":1}
I20260621 02:51:17.842691 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:16.504373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:60695)
0621 02:51:16.505229 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:16.505269 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:16.506987 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:16.508452 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:16.508475 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:16.509050 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:16.509645 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:16.509662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.511194 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.511202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:16.512405 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:16.512416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.513513 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.513520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:16.513788 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:16.514450 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:16.514479 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:17.841402 (+1326923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":343,"threads_started":1}
W20260621 02:51:17.843693 20672 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:45409 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.109.62:45409: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:17.855031 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:17.844454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:36915)
0621 02:51:17.844717 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:17.844739 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:17.844861 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:17.845532 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:17.845539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:17.846095 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:17.846365 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:17.846381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.848153 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.848161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:17.849130 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:17.849141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.850382 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.850390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:17.850562 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.851265 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:17.851282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:17.854745 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260621 02:51:17.855187 20674 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:17.844613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:36915 (local address 127.11.109.62:45409)
0621 02:51:17.844826 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:17.844835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:17.844864 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:51:17.845028 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:17.845036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:17.845127 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:17.845230 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:17.846545 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.847877 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.849369 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.849992 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.850086 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.854943 (+ 4857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:36915: BlockingRecv error: recv got EOF from 127.11.109.1:36915 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:17 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:17.864614 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:17.855919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:48861)
0621 02:51:17.856121 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:17.856139 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:17.856246 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:17.856566 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:17.856571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:17.856871 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:17.857008 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:17.857016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.858035 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.858041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:17.859184 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:17.859195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.859879 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.859884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:17.860050 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.860642 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:17.860659 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:17.864345 (+ 3686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260621 02:51:17.864902 20672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:51:17.864933 20674 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:17.855987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:48861 (local address 127.11.109.62:45409)
0621 02:51:17.856122 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:17.856129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:17.856214 (+ 85us) server_negotiation.cc:407] Connection header received
0621 02:51:17.856367 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:17.856372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:17.856439 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:17.856538 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:17.857140 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.857892 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.859318 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.859711 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.859871 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.864521 (+ 4650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:48861: BlockingRecv error: recv got EOF from 127.11.109.1:48861 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260621 02:51:18.416715 20808 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.428s user 0.010s sys 0.007s
W20260621 02:51:18.417933 20807 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:16.971747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:42681 (local address 127.11.109.62:45409)
0621 02:51:16.978711 (+ 6964us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:16.978716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:16.979370 (+ 654us) server_negotiation.cc:407] Connection header received
0621 02:51:16.981306 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:16.981312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:16.981404 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:16.981524 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:16.983018 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.983749 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.986409 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.987401 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.987614 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.417678 (+1430064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:42681: BlockingRecv error: recv got EOF from 127.11.109.2:42681 (error 108)
Metrics: {"server-negotiator.queue_time_us":6843,"thread_start_us":6720,"threads_started":1}
I20260621 02:51:18.418567 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:16.972724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:42681)
0621 02:51:16.979192 (+ 6468us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:16.979238 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:16.980990 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:16.981770 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:16.981779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:16.982289 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:16.982842 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:16.982859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.983911 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.983919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:16.986182 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:16.986200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:16.987690 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:16.987700 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:16.988040 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:16.988965 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:16.989000 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.417518 (+1428518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6187,"thread_start_us":5936,"threads_started":1}
W20260621 02:51:18.419519 20805 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:45409 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.109.62:45409: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:18.433916 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:18.420333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:49047)
0621 02:51:18.420557 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:18.420572 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:18.420703 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:18.421042 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:18.421047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:18.421517 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:18.421688 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.421696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.422776 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.422782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.423913 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.423925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.424626 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.424632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.424800 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.429486 (+ 4686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:18.429518 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.433623 (+ 4105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260621 02:51:18.434060 20807 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:18.420329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:49047 (local address 127.11.109.62:45409)
0621 02:51:18.420486 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:18.420493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:18.420638 (+ 145us) server_negotiation.cc:407] Connection header received
0621 02:51:18.420850 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:18.420853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:18.420919 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:18.421017 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:18.421819 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.422636 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.424058 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.424455 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.424592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.433817 (+ 9225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:49047: BlockingRecv error: recv got EOF from 127.11.109.2:49047 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:18.446008 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:18.434667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:37097)
0621 02:51:18.434930 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:18.434957 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:18.435128 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:18.435502 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:18.435509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:18.436098 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:18.436391 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.436407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.437474 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.437482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.439758 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.439777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.440515 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.440524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.440794 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.441567 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:18.441602 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.445819 (+ 4217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260621 02:51:18.446269 20805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:51:18.446568 20807 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:18.434822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:37097 (local address 127.11.109.62:45409)
0621 02:51:18.434983 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:18.434987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:18.435027 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:51:18.435265 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:18.435268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:18.435331 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:18.435413 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:18.436558 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.437323 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.439954 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.440341 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.440442 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.446184 (+ 5742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:37097: BlockingRecv error: recv got EOF from 127.11.109.2:37097 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260621 02:51:18.769379 20942 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.273s user 0.003s sys 0.021s
W20260621 02:51:18.770593 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:17.471926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54925 (local address 127.11.109.62:45409)
0621 02:51:17.482286 (+ 10360us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:17.482294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:17.482320 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:51:17.482412 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:17.482421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:17.482522 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:17.482657 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:17.486136 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.487978 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.492965 (+ 4987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.493838 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.493975 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.770299 (+1276324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:54925: BlockingRecv error: recv got EOF from 127.11.109.3:54925 (error 108)
Metrics: {"server-negotiator.queue_time_us":10181,"thread_start_us":5394,"threads_started":1}
I20260621 02:51:18.771164 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:17.471234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:54925)
0621 02:51:17.476433 (+ 5199us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:17.476481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:17.479586 (+ 3105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:17.483733 (+ 4147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:17.483752 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:17.484536 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:17.485443 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:17.485470 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.488279 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.488288 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:17.492726 (+ 4438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:17.492747 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:17.494427 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:17.494437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:17.494843 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:17.495836 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:17.495898 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.770143 (+1274245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4887,"thread_start_us":4652,"threads_started":1}
W20260621 02:51:18.772112 20938 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:45409 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.109.62:45409: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:18.780691 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:18.772839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:54059)
0621 02:51:18.773009 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:18.773028 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:18.773160 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:18.773540 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:18.773545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:18.773891 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:18.774055 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.774064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.774950 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.774953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.775889 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.775899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.776546 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.776550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.776657 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.777302 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:18.777320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.780516 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:51:18.780918 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:18.772890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54059 (local address 127.11.109.62:45409)
0621 02:51:18.773077 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:18.773084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:18.773104 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:51:18.773309 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:18.773313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:18.773385 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:18.773492 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:18.774171 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.774851 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.776004 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.776392 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.776492 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.780698 (+ 4206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:54059: BlockingRecv error: recv got EOF from 127.11.109.3:54059 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:18.789029 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:18.781392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:36377)
0621 02:51:18.781536 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:18.781550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:18.781622 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:18.781944 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:18.781946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:18.782194 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:18.782332 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.782339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.783239 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.783242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.784444 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.784461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.785381 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.785387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.785543 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.786120 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:18.786136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.788868 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 02:51:18.789254 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:18.781453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:36377 (local address 127.11.109.62:45409)
0621 02:51:18.781701 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:18.781704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:18.781718 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:51:18.781752 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:18.781755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:18.781824 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:18.781916 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:18.782446 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.783135 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.784588 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.785237 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.785356 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.789015 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:36377: BlockingRecv error: recv got EOF from 127.11.109.3:36377 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260621 02:51:18.789335 20938 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:18 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:18.874210 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:18.865619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:44919)
0621 02:51:18.866188 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:18.866208 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:18.866355 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:18.866711 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:18.866715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:18.867183 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:18.867324 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.867332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.868317 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.868320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.869285 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:18.869295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.869980 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.869984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:18.870137 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.870681 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:18.870696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:18.873977 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":316,"threads_started":1}
W20260621 02:51:18.874400 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:18.865681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:44919 (local address 127.11.109.62:45409)
0621 02:51:18.866044 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:18.866050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:18.866301 (+ 251us) server_negotiation.cc:407] Connection header received
0621 02:51:18.866481 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:18.866493 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:18.866560 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:18.866670 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:18.867498 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.868198 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.869422 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:18.869835 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:18.869932 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:18.874161 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:44919: BlockingRecv error: recv got EOF from 127.11.109.1:44919 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 02:51:19 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:19 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:19.456825 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:19.446933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:58367)
0621 02:51:19.447439 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:19.447455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:19.447568 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:19.448044 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:19.448049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:19.448614 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:19.448823 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:19.448836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.450209 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.450213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:19.451150 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:19.451161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.451844 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.451848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:19.451957 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:19.452475 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:19.452491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:19.456480 (+ 3989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
W20260621 02:51:19.457214 20947 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:19.447031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:58367 (local address 127.11.109.62:45409)
0621 02:51:19.447772 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:19.447776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:19.447791 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:51:19.447833 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:19.447836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:19.447900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:19.447984 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:19.449000 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.450083 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.451292 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.451695 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.451790 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:19.456739 (+ 4949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:58367: BlockingRecv error: recv got EOF from 127.11.109.2:58367 (error 108)
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":546,"threads_started":1}
Jun 21 02:51:19 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:19 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:19.802030 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:19.789994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:43381)
0621 02:51:19.790495 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:19.790523 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:19.790934 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:19.791268 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:19.791273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:19.791684 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:19.791848 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:19.791858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.793914 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.793919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:19.795427 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:19.795441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.796713 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.796719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:19.796892 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:19.797558 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:19.797584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:19.801631 (+ 4047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":490,"threads_started":1}
W20260621 02:51:19.802140 20947 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:19.790187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:43381 (local address 127.11.109.62:45409)
0621 02:51:19.790386 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:19.790393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:19.790597 (+ 204us) server_negotiation.cc:407] Connection header received
0621 02:51:19.791066 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:19.791071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:19.791146 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:19.791243 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:19.791995 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.793736 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.795589 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.796526 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.796645 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:19.801852 (+ 5207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:43381: BlockingRecv error: recv got EOF from 127.11.109.3:43381 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 21 02:51:19 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:19 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:19.884045 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:19.875159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:51767)
0621 02:51:19.875698 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:19.875714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:19.875826 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:19.876106 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:19.876110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:19.876493 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:19.876665 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:19.876673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.877698 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.877702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:19.878602 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:19.878614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.879323 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.879328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:19.879466 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:19.880015 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:19.880033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:19.883724 (+ 3691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":311,"threads_started":1}
W20260621 02:51:19.884176 20947 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:19.875204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:51767 (local address 127.11.109.62:45409)
0621 02:51:19.875361 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:19.875365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:19.875783 (+ 418us) server_negotiation.cc:407] Connection header received
0621 02:51:19.875915 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:19.875919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:19.875975 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:19.876062 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:19.876780 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.877530 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.878716 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:19.879158 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:19.879247 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:19.883855 (+ 4608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:51767: BlockingRecv error: recv got EOF from 127.11.109.1:51767 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 02:51:20 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:20 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:20.469682 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:20.460662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:38639)
0621 02:51:20.461239 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:20.461259 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:20.461449 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:20.461841 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:20.461845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:20.462465 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:20.462644 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:20.462653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.463647 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.463653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:20.464606 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:20.464618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.465349 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.465355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:20.465545 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:20.466072 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:20.466090 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:20.469442 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":281,"threads_started":1}
W20260621 02:51:20.469869 20951 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:20.460722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:38639 (local address 127.11.109.62:45409)
0621 02:51:20.461263 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:20.461268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:20.461362 (+ 94us) server_negotiation.cc:407] Connection header received
0621 02:51:20.461589 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:20.461595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:20.461659 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:20.461776 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:20.462791 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.463513 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.464745 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.465153 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.465323 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:20.469617 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:38639: BlockingRecv error: recv got EOF from 127.11.109.2:38639 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
Jun 21 02:51:20 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:20 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:20.811352 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:20.803097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:58897)
0621 02:51:20.803541 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:20.803556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:20.803668 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:20.803947 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:20.803950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:20.804335 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:20.804511 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:20.804520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.805486 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.805490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:20.806425 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:20.806435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.807070 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.807074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:20.807188 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:20.807827 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:20.807845 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:20.811125 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":248,"threads_started":1}
W20260621 02:51:20.811523 20951 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:20.803143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:58897 (local address 127.11.109.62:45409)
0621 02:51:20.803357 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:20.803367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:20.803635 (+ 268us) server_negotiation.cc:407] Connection header received
0621 02:51:20.803761 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:20.803764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:20.803825 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:20.803909 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:20.804635 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.805354 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.806547 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.806935 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.807064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:20.811294 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:58897: BlockingRecv error: recv got EOF from 127.11.109.3:58897 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 02:51:20 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:20 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:20.893537 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:20.884947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:57689)
0621 02:51:20.885494 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:20.885510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:20.885622 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:20.885984 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:20.885988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:20.886343 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:20.886513 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:20.886522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.887505 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.887509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:20.888411 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:20.888421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.889056 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.889060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:20.889223 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:20.889783 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:20.889800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:20.893299 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":310,"threads_started":1}
W20260621 02:51:20.893689 20951 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:20.885039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:57689 (local address 127.11.109.62:45409)
0621 02:51:20.885251 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:20.885255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:20.885583 (+ 328us) server_negotiation.cc:407] Connection header received
0621 02:51:20.885714 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:20.885717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:20.885786 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:20.885866 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:20.886639 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.887366 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.888536 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:20.888925 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:20.889010 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:20.893461 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:57689: BlockingRecv error: recv got EOF from 127.11.109.1:57689 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 02:51:21 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:21 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:21.479431 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:21.470596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:33495)
0621 02:51:21.471080 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:21.471105 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:21.471203 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:21.471508 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:21.471511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:21.471822 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:21.471968 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:21.471991 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.473400 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.473406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:21.474418 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:21.474430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.475170 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.475175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:21.475307 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:21.475884 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:21.475902 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:21.479209 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":258,"threads_started":1}
W20260621 02:51:21.479873 20955 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:21.470674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:33495 (local address 127.11.109.62:45409)
0621 02:51:21.471148 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:21.471152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:21.471169 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:51:21.471305 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:21.471308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:21.471380 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:21.471475 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:21.472128 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.473229 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.474539 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.475016 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.475116 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:21.479476 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:33495: BlockingRecv error: recv got EOF from 127.11.109.2:33495 (error 108)
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":268,"threads_started":1}
Jun 21 02:51:21 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:21 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:21.821853 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:21.812452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:54697)
0621 02:51:21.812981 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:21.812999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:21.813141 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:21.813493 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:21.813496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:21.813859 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:21.814039 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:21.814048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.815466 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.815474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:21.816605 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:21.816618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.817321 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.817326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:21.817479 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:21.818018 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:21.818037 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:21.821634 (+ 3597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":246,"threads_started":1}
W20260621 02:51:21.822140 20955 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:21.812642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54697 (local address 127.11.109.62:45409)
0621 02:51:21.812850 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:21.812855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:21.813077 (+ 222us) server_negotiation.cc:407] Connection header received
0621 02:51:21.813309 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:21.813312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:21.813370 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:21.813449 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:21.814194 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.815289 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.816735 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.817109 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.817267 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:21.821825 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:54697: BlockingRecv error: recv got EOF from 127.11.109.3:54697 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 21 02:51:21 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:21 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:21.903237 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:21.894446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:44301)
0621 02:51:21.894995 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:21.895010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:21.895167 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:21.895527 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:21.895531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:21.895993 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:21.896152 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:21.896160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.897292 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.897300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:21.898292 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:21.898302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.899013 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.899019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:21.899192 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:21.899724 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:21.899740 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:21.902895 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":302,"threads_started":1}
W20260621 02:51:21.903311 20955 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:21.894509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:44301 (local address 127.11.109.62:45409)
0621 02:51:21.894816 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:21.894822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:21.895098 (+ 276us) server_negotiation.cc:407] Connection header received
0621 02:51:21.895288 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:21.895293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:21.895385 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:21.895491 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:21.896290 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.897132 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.898440 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:21.898867 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:21.899001 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:21.903083 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:44301: BlockingRecv error: recv got EOF from 127.11.109.1:44301 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Jun 21 02:51:22 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:22 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:22.489085 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:22.480398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:37611)
0621 02:51:22.480914 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:22.480932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:22.481039 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:22.481466 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:22.481469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:22.481871 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:22.482034 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:22.482044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.483028 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.483032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:22.484029 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:22.484041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.484757 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.484763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:22.484949 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:22.485670 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:22.485698 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:22.488833 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":283,"threads_started":1}
W20260621 02:51:22.489398 20959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:22.480450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:37611 (local address 127.11.109.62:45409)
0621 02:51:22.480935 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:22.480939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:22.480983 (+ 44us) server_negotiation.cc:407] Connection header received
0621 02:51:22.481150 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:22.481154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:22.481261 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:22.481419 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:22.482158 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.482895 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.484167 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.484598 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.484740 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:22.489020 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:37611: BlockingRecv error: recv got EOF from 127.11.109.2:37611 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":300,"threads_started":1}
Jun 21 02:51:22 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:22 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:22.831426 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:22.822763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:33141)
0621 02:51:22.823203 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:22.823219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:22.823330 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:22.823595 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:22.823598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:22.823970 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:22.824121 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:22.824130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.825076 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.825082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:22.826412 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:22.826426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.827129 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.827134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:22.827304 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:22.827926 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:22.827946 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:22.831173 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":271,"threads_started":1}
W20260621 02:51:22.831598 20959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:22.822793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:33141 (local address 127.11.109.62:45409)
0621 02:51:22.822994 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:22.823002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:22.823295 (+ 293us) server_negotiation.cc:407] Connection header received
0621 02:51:22.823419 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:22.823423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:22.823478 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:22.823560 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:22.824236 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.824938 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.826582 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.826979 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.827107 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:22.831360 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:33141: BlockingRecv error: recv got EOF from 127.11.109.3:33141 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 21 02:51:22 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:22 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:22.913116 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:22.904135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:50877)
0621 02:51:22.904633 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:22.904648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:22.904758 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:22.905037 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:22.905040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:22.905513 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:22.905661 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:22.905670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.906770 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.906775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:22.907922 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:22.907942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.908719 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.908725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:22.908912 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:22.909470 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:22.909487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:22.912806 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
W20260621 02:51:22.913267 20959 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:22.904325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:50877 (local address 127.11.109.62:45409)
0621 02:51:22.904490 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:22.904494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:22.904723 (+ 229us) server_negotiation.cc:407] Connection header received
0621 02:51:22.904844 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:22.904848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:22.904918 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:22.904997 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:22.905775 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.906644 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.908073 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:22.908577 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:22.908709 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:22.913005 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:50877: BlockingRecv error: recv got EOF from 127.11.109.1:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 02:51:23 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:23 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:23.503620 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:23.489898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:42003)
0621 02:51:23.490520 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:23.490536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:23.490636 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:23.490953 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:23.490957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:23.491372 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:23.491514 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:23.491522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.492516 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.492519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:23.493523 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:23.493535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.494344 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.494350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:23.494563 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:23.499731 (+ 5168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:23.499749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:23.503413 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":382,"threads_started":1}
W20260621 02:51:23.503834 20963 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:23.490057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:42003 (local address 127.11.109.62:45409)
0621 02:51:23.490592 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:23.490596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:23.490611 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:51:23.490747 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:23.490751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:23.490806 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:23.490890 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:23.491696 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.492389 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.493647 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.494171 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.494308 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:23.503605 (+ 9297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:42003: BlockingRecv error: recv got EOF from 127.11.109.2:42003 (error 108)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":321,"threads_started":1}
Jun 21 02:51:23 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:23 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:23.841578 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:23.832369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:40091)
0621 02:51:23.832981 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:23.832999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:23.833109 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:23.833474 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:23.833478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:23.833884 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:23.834060 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:23.834069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.835274 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.835278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:23.836448 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:23.836463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.837243 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.837248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:23.837374 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:23.837906 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:23.837923 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:23.841223 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":336,"threads_started":1}
W20260621 02:51:23.841688 20963 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:23.832439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:40091 (local address 127.11.109.62:45409)
0621 02:51:23.832656 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:23.832661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:23.833089 (+ 428us) server_negotiation.cc:407] Connection header received
0621 02:51:23.833274 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:23.833278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:23.833339 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:23.833448 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:23.834201 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.835128 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.836622 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.837063 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.837161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:23.841454 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:40091: BlockingRecv error: recv got EOF from 127.11.109.3:40091 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 21 02:51:23 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:23 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:23.923012 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:23.914020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:50729)
0621 02:51:23.914551 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:23.914566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:23.914718 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:23.915064 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:23.915068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:23.915470 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:23.915636 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:23.915644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.916674 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.916681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:23.917784 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:23.917818 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.918766 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.918772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:23.918976 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:23.919515 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:23.919531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:23.922709 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":299,"threads_started":1}
W20260621 02:51:23.923087 20963 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:23.914155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:50729 (local address 127.11.109.62:45409)
0621 02:51:23.914383 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:23.914389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:23.914651 (+ 262us) server_negotiation.cc:407] Connection header received
0621 02:51:23.914834 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:23.914840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:23.914921 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:23.915032 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:23.915774 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.916537 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.917947 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:23.918605 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:23.918727 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:23.922879 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:50729: BlockingRecv error: recv got EOF from 127.11.109.1:50729 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 21 02:51:24 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:24 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:24.513293 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:24.504593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:32901)
0621 02:51:24.505161 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:24.505183 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:24.505329 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:24.505744 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:24.505748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:24.506152 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:24.506317 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:24.506325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.507341 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.507348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:24.508259 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:24.508269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.508982 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.508987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:24.509163 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:24.509726 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:24.509742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:24.513020 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
W20260621 02:51:24.513510 20967 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:24.504652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:32901 (local address 127.11.109.62:45409)
0621 02:51:24.505449 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:24.505455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:24.505473 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:51:24.505525 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:24.505530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:24.505616 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:24.505718 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:24.506482 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.507200 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.508388 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.508827 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.508955 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:24.513284 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:32901: BlockingRecv error: recv got EOF from 127.11.109.2:32901 (error 108)
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":537,"threads_started":1}
Jun 21 02:51:24 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:24 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:24.851231 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:24.842537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:50739)
0621 02:51:24.843034 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:24.843049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:24.843170 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:24.843438 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:24.843441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:24.843824 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:24.843996 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:24.844004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.844940 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.844944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:24.845871 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:24.845882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.846615 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.846621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:24.846783 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:24.847388 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:24.847406 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:24.850895 (+ 3489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":291,"threads_started":1}
W20260621 02:51:24.851315 20967 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:24.842586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:50739 (local address 127.11.109.62:45409)
0621 02:51:24.842811 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:24.842820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:24.843141 (+ 321us) server_negotiation.cc:407] Connection header received
0621 02:51:24.843262 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:24.843265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:24.843324 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:24.843398 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:24.844119 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.844823 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.846022 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.846446 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.846584 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:24.851078 (+ 4494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:50739: BlockingRecv error: recv got EOF from 127.11.109.3:50739 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 02:51:24 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:24 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:24.932699 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:24.923848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:51541)
0621 02:51:24.924357 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:24.924374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:24.924484 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:24.924812 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:24.924815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:24.925265 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:24.925436 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:24.925444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.926433 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.926438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:24.927405 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:24.927418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.928057 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.928061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:24.928170 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:24.928851 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:24.928874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:24.932482 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":256,"threads_started":1}
W20260621 02:51:24.932834 20967 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:24.923967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:51541 (local address 127.11.109.62:45409)
0621 02:51:24.924201 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:24.924208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:24.924451 (+ 243us) server_negotiation.cc:407] Connection header received
0621 02:51:24.924602 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:24.924605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:24.924663 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:24.924756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:24.925551 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.926287 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.927537 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:24.927932 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:24.928015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:24.932626 (+ 4611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:51541: BlockingRecv error: recv got EOF from 127.11.109.1:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 02:51:25 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:25 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:25.522912 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:25.514208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:41853)
0621 02:51:25.514892 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:25.514908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:25.515090 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:25.515492 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:25.515498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:25.515916 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:25.516088 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:25.516096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.517035 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.517039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:25.518003 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:25.518013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.518643 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.518647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:25.518750 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:25.519258 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:25.519274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:25.522658 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":324,"threads_started":1}
W20260621 02:51:25.523232 20970 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:25.514267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:41853 (local address 127.11.109.62:45409)
0621 02:51:25.514805 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:25.514811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:25.515035 (+ 224us) server_negotiation.cc:407] Connection header received
0621 02:51:25.515219 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:25.515226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:25.515326 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:25.515448 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:25.516205 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.516919 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.518114 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.518518 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.518605 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:25.522968 (+ 4363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:41853: BlockingRecv error: recv got EOF from 127.11.109.2:41853 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":338,"threads_started":1}
Jun 21 02:51:25 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:25 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:25.860702 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:25.852107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:52753)
0621 02:51:25.852646 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:25.852662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:25.852768 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:25.853038 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:25.853041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:25.853467 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:25.853656 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:25.853665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.854599 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.854602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:25.855678 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:25.855692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.856501 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.856509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:25.856674 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:25.857333 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:25.857350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:25.860394 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
W20260621 02:51:25.860798 20970 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:25.852215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:52753 (local address 127.11.109.62:45409)
0621 02:51:25.852379 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:25.852383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:25.852734 (+ 351us) server_negotiation.cc:407] Connection header received
0621 02:51:25.852856 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:25.852859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:25.852915 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:25.853007 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:25.853771 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.854492 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.855827 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.856334 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.856462 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:25.860587 (+ 4125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:52753: BlockingRecv error: recv got EOF from 127.11.109.3:52753 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 02:51:25 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:25 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:25.942338 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:25.933634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:54055)
0621 02:51:25.934150 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:25.934166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:25.934271 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:25.934560 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:25.934563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:25.934932 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:25.935088 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:25.935096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.936015 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.936019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:25.937083 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:25.937100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.937952 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.937958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:25.938150 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:25.938829 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:25.938845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:25.941992 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
W20260621 02:51:25.942456 20970 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:25.933806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:54055 (local address 127.11.109.62:45409)
0621 02:51:25.933954 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:25.933958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:25.934235 (+ 277us) server_negotiation.cc:407] Connection header received
0621 02:51:25.934358 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:25.934362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:25.934428 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:25.934515 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:25.935201 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.935894 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.937229 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:25.937777 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:25.937913 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:25.942194 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:54055: BlockingRecv error: recv got EOF from 127.11.109.1:54055 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 02:51:26 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:26 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:26.533048 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:26.523897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:37807)
0621 02:51:26.524685 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:26.524723 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:26.524853 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:26.525129 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:26.525132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:26.525568 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:26.525738 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:26.525746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.526825 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.526829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:26.527738 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:26.527748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.528422 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.528429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:26.528572 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:26.529298 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:26.529315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:26.532792 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":287,"threads_started":1}
W20260621 02:51:26.533365 20975 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:26.523968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:37807 (local address 127.11.109.62:45409)
0621 02:51:26.524763 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:26.524768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:26.524782 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:51:26.524937 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:26.524940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:26.524995 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:26.525079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:26.525866 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.526706 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.527852 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.528243 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.528336 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:26.532994 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:37807: BlockingRecv error: recv got EOF from 127.11.109.2:37807 (error 108)
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":528,"threads_started":1}
Jun 21 02:51:26 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:26 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:26.870558 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:26.861723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:50295)
0621 02:51:26.862248 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:26.862263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:26.862419 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:26.862744 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:26.862749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:26.863193 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:26.863348 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:26.863357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.864359 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.864365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:26.865367 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:26.865378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.866091 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.866097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:26.866263 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:26.866830 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:26.866848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:26.870317 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":285,"threads_started":1}
W20260621 02:51:26.870711 20975 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:26.861814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:50295 (local address 127.11.109.62:45409)
0621 02:51:26.862013 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:26.862021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:26.862352 (+ 331us) server_negotiation.cc:407] Connection header received
0621 02:51:26.862542 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:26.862551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:26.862609 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:26.862716 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:26.863491 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.864233 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.865509 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.865914 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.866055 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:26.870485 (+ 4430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:50295: BlockingRecv error: recv got EOF from 127.11.109.3:50295 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 02:51:26 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:26 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:26.952284 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:26.943274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:56011)
0621 02:51:26.943797 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:26.943813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:26.943917 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:26.944300 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:26.944303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:26.944708 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:26.944850 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:26.944858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.945840 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.945843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:26.946843 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:26.946858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.947648 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.947653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:26.947801 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:26.948474 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:26.948491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:26.952063 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
W20260621 02:51:26.952484 20975 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:26.943411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:56011 (local address 127.11.109.62:45409)
0621 02:51:26.943579 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:26.943583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:26.943917 (+ 334us) server_negotiation.cc:407] Connection header received
0621 02:51:26.944014 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:26.944018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:26.944104 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:26.944195 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:26.944986 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.945724 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.947035 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:26.947524 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:26.947628 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:26.952240 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:56011: BlockingRecv error: recv got EOF from 127.11.109.1:56011 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 02:51:27 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:27 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:27.543154 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:27.533957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:39981)
0621 02:51:27.534455 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:27.534470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:27.534566 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:27.534993 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:27.534996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:27.535385 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:27.535521 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:27.535529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.536505 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.536509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:27.537449 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:27.537459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.538140 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.538144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:27.538274 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:27.538961 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:27.538979 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:27.542834 (+ 3855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":225,"threads_started":1}
W20260621 02:51:27.543279 20979 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:27.534005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:39981 (local address 127.11.109.62:45409)
0621 02:51:27.534734 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:27.534738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:27.534752 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:51:27.534804 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:27.534808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:27.534865 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:27.534950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:27.535652 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.536383 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.537564 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.537990 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.538090 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:27.543020 (+ 4930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:39981: BlockingRecv error: recv got EOF from 127.11.109.2:39981 (error 108)
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":551,"threads_started":1}
Jun 21 02:51:27 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:27 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:27.895756 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:27.871606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:58241)
0621 02:51:27.881639 (+ 10033us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:27.881664 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:27.881865 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:27.882291 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:27.882298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:27.882905 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:27.883114 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:27.883127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.884745 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.884754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:27.886488 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:27.886505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.887585 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.887593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:27.887814 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:27.888535 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:27.888563 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:27.895313 (+ 6750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9877,"thread_start_us":9736,"threads_started":1}
W20260621 02:51:27.895844 20979 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:27.871673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:58241 (local address 127.11.109.62:45409)
0621 02:51:27.871892 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:27.871901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:27.881787 (+ 9886us) server_negotiation.cc:407] Connection header received
0621 02:51:27.882010 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:27.882018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:27.882116 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:27.882245 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:27.883280 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.884587 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.886658 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.887406 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.887569 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:27.895546 (+ 7977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:58241: BlockingRecv error: recv got EOF from 127.11.109.3:58241 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 21 02:51:27 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:27 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:27.962021 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:27.953239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:60809)
0621 02:51:27.953780 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:27.953795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:27.953902 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:27.954188 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:27.954191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:27.954587 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:27.954740 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:27.954749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.955691 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.955695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:27.956678 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:27.956689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.957364 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.957368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:27.957482 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:27.958041 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:27.958057 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:27.961772 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
W20260621 02:51:27.962102 20979 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:27.953285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:60809 (local address 127.11.109.62:45409)
0621 02:51:27.953416 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:27.953421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:27.953864 (+ 443us) server_negotiation.cc:407] Connection header received
0621 02:51:27.953998 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:27.954002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:27.954069 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:27.954146 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:27.954859 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.955579 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.956801 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:27.957238 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:27.957328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:27.961914 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:60809: BlockingRecv error: recv got EOF from 127.11.109.1:60809 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 21 02:51:28 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:28 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:28.555917 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:28.544017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:51793)
0621 02:51:28.544531 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:28.544552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:28.544693 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:28.544995 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:28.544998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:28.545366 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:28.545507 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:28.545515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.546606 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.546609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:28.547652 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:28.547663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.548469 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.548473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:28.548586 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:28.551854 (+ 3268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:28.551872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:28.555686 (+ 3814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":282,"threads_started":1}
W20260621 02:51:28.556232 20983 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:28.544062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:51793 (local address 127.11.109.62:45409)
0621 02:51:28.544683 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:28.544706 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:28.544730 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:51:28.544793 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:28.544796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:28.544855 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:28.544948 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:28.545662 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.546470 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.547844 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.548340 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.548442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:28.555920 (+ 7478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:51793: BlockingRecv error: recv got EOF from 127.11.109.2:51793 (error 108)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":398,"threads_started":1}
Jun 21 02:51:28 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:28 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:28.905388 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:28.896751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:49887)
0621 02:51:28.897349 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:28.897366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:28.897477 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:28.897763 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:28.897766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:28.898136 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:28.898289 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:28.898297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.899269 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.899272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:28.900213 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:28.900226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.900850 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.900854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:28.900972 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:28.901561 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:28.901578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:28.905014 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":317,"threads_started":1}
W20260621 02:51:28.905525 20983 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:28.896951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:49887 (local address 127.11.109.62:45409)
0621 02:51:28.897131 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:28.897135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:28.897438 (+ 303us) server_negotiation.cc:407] Connection header received
0621 02:51:28.897569 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:28.897573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:28.897631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:28.897708 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:28.898406 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.899156 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.900332 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.900704 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.900790 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:28.905289 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:49887: BlockingRecv error: recv got EOF from 127.11.109.3:49887 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 02:51:28 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:28 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:28.971721 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:28.963159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:44183)
0621 02:51:28.963715 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:28.963735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:28.963877 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:28.964227 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:28.964232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:28.964651 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:28.964814 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:28.964822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.965867 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.965874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:28.966793 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:28.966804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.967495 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.967501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:28.967681 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:28.968293 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:28.968309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:28.971488 (+ 3179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
W20260621 02:51:28.971909 20983 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:28.963216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:44183 (local address 127.11.109.62:45409)
0621 02:51:28.963416 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:28.963425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:28.963804 (+ 379us) server_negotiation.cc:407] Connection header received
0621 02:51:28.964002 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:28.964007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:28.964087 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:28.964201 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:28.964949 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.965735 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.966951 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:28.967350 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:28.967479 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:28.971676 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:44183: BlockingRecv error: recv got EOF from 127.11.109.1:44183 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 02:51:29 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:29 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:29.565984 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:29.556804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:57399)
0621 02:51:29.557369 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:29.557386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:29.557486 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:29.557792 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:29.557795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:29.558196 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:29.558363 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:29.558371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.559401 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.559404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:29.560308 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:29.560319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.561372 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.561380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:29.561591 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:29.562243 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:29.562260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:29.565775 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":307,"threads_started":1}
W20260621 02:51:29.566176 20987 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:29.556942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:57399 (local address 127.11.109.62:45409)
0621 02:51:29.557452 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:29.557456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:29.557472 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:51:29.557604 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:29.557608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:29.557665 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:29.557752 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:29.558495 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.559276 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.560429 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.561178 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.561328 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:29.565944 (+ 4616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:57399: BlockingRecv error: recv got EOF from 127.11.109.2:57399 (error 108)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":291,"threads_started":1}
Jun 21 02:51:29 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:29 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:29.914661 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:29.906314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:58445)
0621 02:51:29.906879 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:29.906896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:29.907005 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:29.907291 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:29.907294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:29.907680 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:29.907853 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:29.907861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.908804 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.908808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:29.909780 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:29.909792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.910396 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.910400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:29.910512 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:29.911058 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:29.911076 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:29.914439 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":285,"threads_started":1}
W20260621 02:51:29.914800 20987 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:29.906376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:58445 (local address 127.11.109.62:45409)
0621 02:51:29.906579 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:29.906584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:29.906969 (+ 385us) server_negotiation.cc:407] Connection header received
0621 02:51:29.907101 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:29.907104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:29.907170 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:29.907248 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:29.907970 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.908690 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.909903 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.910284 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.910369 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:29.914592 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:58445: BlockingRecv error: recv got EOF from 127.11.109.3:58445 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 02:51:29 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:29 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:29.981462 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:29.972646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:33389)
0621 02:51:29.973151 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:29.973167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:29.973298 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:29.973592 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:29.973596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:29.974017 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:29.974172 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:29.974180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.975259 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.975266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:29.976175 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:29.976185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.976939 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.976945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:29.977117 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:29.977763 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:29.977780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:29.981135 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":289,"threads_started":1}
W20260621 02:51:29.981659 20987 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:29.972821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:33389 (local address 127.11.109.62:45409)
0621 02:51:29.973058 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:29.973064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:29.973261 (+ 197us) server_negotiation.cc:407] Connection header received
0621 02:51:29.973395 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:29.973399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:29.973474 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:29.973557 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:29.974325 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.975104 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.976314 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:29.976744 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:29.976885 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:29.981430 (+ 4545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:33389: BlockingRecv error: recv got EOF from 127.11.109.1:33389 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 21 02:51:30 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:30 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:30.575870 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:30.566792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:46917)
0621 02:51:30.567304 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:30.567327 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:30.567452 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:30.567882 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:30.567889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:30.568449 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:30.568620 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:30.568629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.569802 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.569806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:30.570705 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:30.570716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.571362 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.571366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:30.571491 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:30.572125 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:30.572143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:30.575626 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":284,"threads_started":1}
W20260621 02:51:30.576061 20991 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:30.566870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:46917 (local address 127.11.109.62:45409)
0621 02:51:30.567396 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:30.567404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:30.567424 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:51:30.567581 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:30.567589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:30.567679 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:30.567813 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:30.568749 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.569678 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.570820 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.571247 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.571337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:30.575807 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:46917: BlockingRecv error: recv got EOF from 127.11.109.2:46917 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
Jun 21 02:51:30 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:30 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:30.924103 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:30.915511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:33581)
0621 02:51:30.916031 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:30.916047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:30.916151 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:30.916415 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:30.916417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:30.916922 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:30.917082 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:30.917091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.918145 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.918148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:30.919041 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:30.919051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.919661 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.919665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:30.919795 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:30.920318 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:30.920334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:30.923768 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":313,"threads_started":1}
W20260621 02:51:30.924178 20991 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:30.915596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:33581 (local address 127.11.109.62:45409)
0621 02:51:30.915748 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:30.915753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:30.916114 (+ 361us) server_negotiation.cc:407] Connection header received
0621 02:51:30.916245 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:30.916249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:30.916303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:30.916378 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:30.917218 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.918022 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.919155 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.919533 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.919620 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:30.923951 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:33581: BlockingRecv error: recv got EOF from 127.11.109.3:33581 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 02:51:30 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:30 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:30.991065 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:30.982406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:50661)
0621 02:51:30.982839 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:30.982856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:30.982970 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:30.983258 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:30.983262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:30.983632 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:30.983776 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:30.983784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.985236 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.985243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:30.986228 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:30.986240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.986869 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.986873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:30.986986 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:30.987523 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:30.987539 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:30.990835 (+ 3296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
W20260621 02:51:30.991367 20991 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:30.982488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:50661 (local address 127.11.109.62:45409)
0621 02:51:30.982706 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:30.982712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:30.982944 (+ 232us) server_negotiation.cc:407] Connection header received
0621 02:51:30.983058 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:30.983061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:30.983135 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:30.983215 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:30.983924 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.985063 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.986352 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:30.986751 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:30.986847 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:30.991001 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:50661: BlockingRecv error: recv got EOF from 127.11.109.1:50661 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 02:51:31 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:31 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.2@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:31.585626 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:31.576828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.2:60703)
0621 02:51:31.577403 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:31.577420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:31.577520 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:31.577786 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:31.577789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:31.578162 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:31.578311 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:31.578319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.579235 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.579239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:31.580203 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:31.580244 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.580948 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.580952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:31.581089 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:31.581806 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:31.581824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:31.585407 (+ 3583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":350,"threads_started":1}
W20260621 02:51:31.585770 20995 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:31.576961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:60703 (local address 127.11.109.62:45409)
0621 02:51:31.577494 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:31.577498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:31.577514 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:51:31.577618 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:31.577622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:31.577682 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:31.577766 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:31.578425 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.579102 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.580387 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.580822 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.580914 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:31.585552 (+ 4638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.2:60703: BlockingRecv error: recv got EOF from 127.11.109.2:60703 (error 108)
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":312,"threads_started":1}
Jun 21 02:51:31 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:31 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.3@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:31.933670 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:31.925162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.3:60957)
0621 02:51:31.925657 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:31.925673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:31.925819 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:31.926156 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:31.926161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:31.926564 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:31.926708 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:31.926717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.927773 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.927782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:31.928658 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:31.928668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.929427 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.929434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:31.929628 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:31.930281 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:31.930300 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:31.933421 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":239,"threads_started":1}
W20260621 02:51:31.933903 20995 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:31.925283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:60957 (local address 127.11.109.62:45409)
0621 02:51:31.925504 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:31.925510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:31.925755 (+ 245us) server_negotiation.cc:407] Connection header received
0621 02:51:31.925950 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:31.925954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:31.926013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:31.926124 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:31.926849 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.927611 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.928808 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.929273 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.929414 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:31.933602 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.3:60957: BlockingRecv error: recv got EOF from 127.11.109.3:60957 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 21 02:51:31 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:51:31 dist-test-slave-bc52 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.109.1@KRBTEST.COM for oryx/127.11.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:51:32.001168 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:31.992178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45409 (local address 127.11.109.1:59029)
0621 02:51:31.992676 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:31.992702 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:31.992810 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:31.993103 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:31.993106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:31.993495 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:31.993671 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:31.993679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.994739 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.994744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:31.995643 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:31.995654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.996295 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.996299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:31.996428 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:31.997120 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:31.997146 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:32.000806 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.109.62:45409: Server oryx/127.11.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":278,"threads_started":1}
W20260621 02:51:32.001240 20995 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:31.992231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:59029 (local address 127.11.109.62:45409)
0621 02:51:31.992553 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:31.992559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:31.992784 (+ 225us) server_negotiation.cc:407] Connection header received
0621 02:51:31.992905 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:31.992908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:31.992979 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:31.993059 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:31.993831 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.994583 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.995786 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:31.996174 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:31.996267 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:32.000983 (+ 4716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.109.1:59029: BlockingRecv error: recv got EOF from 127.11.109.1:59029 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
I20260621 02:51:32.466634 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20525
I20260621 02:51:32.478574 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:32.467805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.11.109.1:42783)
0621 02:51:32.468494 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:32.468501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:32.468529 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:51:32.468584 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:32.468590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:32.468818 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:32.468972 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:32.470410 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:32.471443 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:32.472569 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:32.473096 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:32.473218 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:32.475623 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:32.475631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:32.475644 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:32.475681 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:32.478036 (+ 2355us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:51:32.478318 (+ 282us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52884: Request ticket server kudu/127.11.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":388,"threads_started":1}
W20260621 02:51:32.478675 20999 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52884: Request ticket server kudu/127.11.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:51:32.478718 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20525: Not authorized: Client connection negotiation failed: client connection to 127.11.109.1:42783: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:51:32.478958 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20525
I20260621 02:51:32.496582 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20676
I20260621 02:51:32.509970 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:32.497692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.11.109.2:42557)
0621 02:51:32.498188 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:32.498195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:32.498231 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:51:32.498290 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:32.498296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:32.498532 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:32.498742 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:32.500064 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:32.501352 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:32.503640 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:32.504189 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:32.504327 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:32.506788 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:32.506797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:32.506810 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:32.506853 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:32.509147 (+ 2294us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:51:32.509650 (+ 503us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49488: Request ticket server kudu/127.11.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
W20260621 02:51:32.509963 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20676: Not authorized: Client connection negotiation failed: client connection to 127.11.109.2:42557: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:51:32.510113 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20676
W20260621 02:51:32.510093 21000 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49488: Request ticket server kudu/127.11.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:51:32.527068 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20809
I20260621 02:51:32.539304 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:32.528125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.11.109.3:40943)
0621 02:51:32.528827 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:32.528834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:32.528858 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:51:32.528917 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:32.528923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:32.529169 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:32.529421 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:32.530772 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:32.531774 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:32.532919 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:32.533710 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:32.533846 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:32.536197 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:32.536207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:32.536233 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:32.536268 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:32.538614 (+ 2346us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:51:32.538951 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51132: Request ticket server kudu/127.11.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":380,"threads_started":1}
W20260621 02:51:32.539278 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20809: Not authorized: Client connection negotiation failed: client connection to 127.11.109.3:40943: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:51:32.539398 21001 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51132: Request ticket server kudu/127.11.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:51:32.539443 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20809
I20260621 02:51:32.555395 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 20456
I20260621 02:51:32.556648 20518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:32.674333 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 20456
2026-06-21T02:51:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20988 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T02:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:32Z Disabled control of system clock
I20260621 02:51:32.747071 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:34927
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:34927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 02:51:33.060985 21006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:33.061266 21006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:33.061393 21006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:33.068442 21006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:33.068533 21006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:33.068631 21006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:33.068733 21006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:33.068805 21006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:33.077575 21006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:34927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:33.078768 21006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:33.080480 21006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:33.091852 21012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:33.091964 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:33.091895 21014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:33.092897 21006 server_base.cc:1034] running on GCE node
I20260621 02:51:33.093729 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:33.095043 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:33.096264 21006 hybrid_clock.cc:648] HybridClock initialized: now 1782010293096228 us; error 53 us; skew 500 ppm
I20260621 02:51:33.096632 21006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:33.098843 21006 webserver.cc:460] Webserver started at http://127.11.109.62:43993/ using document root <none> and password file <none>
I20260621 02:51:33.099519 21006 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:33.099659 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:33.099975 21006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:33.102804 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "6825855277bb434499cb1c8c6412e5aa"
format_stamp: "Formatted at 2026-06-21 02:51:33 on dist-test-slave-bc52"
server_key: "c0c495a2b1c1a2d0c68a0a2e60d59a22"
server_key_iv: "89e59023b4d125c0be42deb21a92dff1"
server_key_version: "encryptionkey@0"
I20260621 02:51:33.103649 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "6825855277bb434499cb1c8c6412e5aa"
format_stamp: "Formatted at 2026-06-21 02:51:33 on dist-test-slave-bc52"
server_key: "c0c495a2b1c1a2d0c68a0a2e60d59a22"
server_key_iv: "89e59023b4d125c0be42deb21a92dff1"
server_key_version: "encryptionkey@0"
I20260621 02:51:33.108470 21006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260621 02:51:33.111857 21020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:33.113173 21006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:51:33.113369 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "6825855277bb434499cb1c8c6412e5aa"
format_stamp: "Formatted at 2026-06-21 02:51:33 on dist-test-slave-bc52"
server_key: "c0c495a2b1c1a2d0c68a0a2e60d59a22"
server_key_iv: "89e59023b4d125c0be42deb21a92dff1"
server_key_version: "encryptionkey@0"
I20260621 02:51:33.113569 21006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:33.124639 21006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:33.125402 21006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:33.125684 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:33.143940 21006 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:34927
I20260621 02:51:33.143958 21071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:34927 every 8 connection(s)
I20260621 02:51:33.145215 21006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:51:33.148676 21072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:33.154798 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21006
I20260621 02:51:33.154990 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:51:33.155833 11700 external_mini_cluster.cc:1428] Setting key eaeebf889beb88faeca020044affb008
I20260621 02:51:33.159763 21072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa: Bootstrap starting.
I20260621 02:51:33.162823 21072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:33.163808 21072 log.cc:824] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:33.166564 21072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa: No bootstrap required, opened a new log
I20260621 02:51:33.169731 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:33.157994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.11.109.62:34927)
0621 02:51:33.158905 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:33.158916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:33.158953 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:51:33.160656 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:33.160679 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:33.161134 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:33.161575 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:33.162137 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:33.163478 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:33.165292 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:33.165966 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:33.166205 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:33.167423 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:33.167430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:33.167443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:33.167971 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:33.168055 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:33.168078 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:33.168266 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:33.168481 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 02:51:33.168758 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":316,"threads_started":1}
I20260621 02:51:33.174540 21072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6825855277bb434499cb1c8c6412e5aa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34927 } }
I20260621 02:51:33.174870 21072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:33.174991 21072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6825855277bb434499cb1c8c6412e5aa, State: Initialized, Role: FOLLOWER
I20260621 02:51:33.175493 21072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6825855277bb434499cb1c8c6412e5aa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34927 } }
I20260621 02:51:33.175720 21072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:51:33.175863 21072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:51:33.176033 21072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:33.178200 21072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6825855277bb434499cb1c8c6412e5aa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34927 } }
I20260621 02:51:33.178682 21072 leader_election.cc:302] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6825855277bb434499cb1c8c6412e5aa; no voters:
I20260621 02:51:33.179250 21072 leader_election.cc:288] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:51:33.179338 21077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:33.180050 21077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [term 1 LEADER]: Becoming Leader. State: Replica: 6825855277bb434499cb1c8c6412e5aa, State: Running, Role: LEADER
I20260621 02:51:33.180577 21077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6825855277bb434499cb1c8c6412e5aa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34927 } }
I20260621 02:51:33.181224 21072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:51:33.183283 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6825855277bb434499cb1c8c6412e5aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6825855277bb434499cb1c8c6412e5aa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34927 } } }
I20260621 02:51:33.183601 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:33.184650 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6825855277bb434499cb1c8c6412e5aa. Latest consensus state: current_term: 1 leader_uuid: "6825855277bb434499cb1c8c6412e5aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6825855277bb434499cb1c8c6412e5aa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34927 } } }
I20260621 02:51:33.184983 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:33.186167 21084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:33.191401 21084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:33.199081 21084 catalog_manager.cc:1347] Generated new cluster ID: 6758d990b5db4166b5e6d963cfcd1ed0
I20260621 02:51:33.199173 21084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:33.221386 21084 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:51:33.222620 21084 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:33.228605 21084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6825855277bb434499cb1c8c6412e5aa: Generated new TSK 0
I20260621 02:51:33.229399 21084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:51:33.241093 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34927
--builtin_ntp_servers=127.11.109.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 02:51:33.578135 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:33.578384 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:33.578465 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:33.585600 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:33.585685 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:33.585795 21096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:51:33.594075 21096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:33.595182 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:33.596777 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:33.608835 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:33.608880 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:33.608835 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:33.609911 21096 server_base.cc:1034] running on GCE node
I20260621 02:51:33.610558 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:33.611483 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:33.612679 21096 hybrid_clock.cc:648] HybridClock initialized: now 1782010293612641 us; error 39 us; skew 500 ppm
I20260621 02:51:33.612988 21096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:33.619436 21096 webserver.cc:460] Webserver started at http://127.11.109.1:36301/ using document root <none> and password file <none>
I20260621 02:51:33.620113 21096 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:33.620223 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:33.620545 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:33.623292 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "55f6750dc3924b7fb8a685ca7477aa4a"
format_stamp: "Formatted at 2026-06-21 02:51:33 on dist-test-slave-bc52"
server_key: "24087d2b4ee8335bd8ccb29b12a1d4a3"
server_key_iv: "ed6039f90c0b081e9cf06af9c2a43259"
server_key_version: "encryptionkey@0"
I20260621 02:51:33.624047 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "55f6750dc3924b7fb8a685ca7477aa4a"
format_stamp: "Formatted at 2026-06-21 02:51:33 on dist-test-slave-bc52"
server_key: "24087d2b4ee8335bd8ccb29b12a1d4a3"
server_key_iv: "ed6039f90c0b081e9cf06af9c2a43259"
server_key_version: "encryptionkey@0"
I20260621 02:51:33.628921 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260621 02:51:33.632093 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:33.633378 21096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:51:33.633534 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "55f6750dc3924b7fb8a685ca7477aa4a"
format_stamp: "Formatted at 2026-06-21 02:51:33 on dist-test-slave-bc52"
server_key: "24087d2b4ee8335bd8ccb29b12a1d4a3"
server_key_iv: "ed6039f90c0b081e9cf06af9c2a43259"
server_key_version: "encryptionkey@0"
I20260621 02:51:33.633739 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:33.658337 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:33.659117 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:33.659406 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:33.660300 21096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:33.661693 21096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:33.661779 21096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:33.661926 21096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:33.662025 21096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:33.694382 21096 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:46547
I20260621 02:51:33.694486 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:46547 every 8 connection(s)
I20260621 02:51:33.695627 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:51:33.705343 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21096
I20260621 02:51:33.705521 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:51:33.706151 11700 external_mini_cluster.cc:1428] Setting key 0e22570164c21971f2e698b1388bfe89
I20260621 02:51:33.707871 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:33.697967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:57475 (local address 127.11.109.62:34927)
0621 02:51:33.698401 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:33.698405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:33.699413 (+ 1008us) server_negotiation.cc:407] Connection header received
0621 02:51:33.701144 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:33.701150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:33.701271 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:33.701389 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:33.702414 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:33.703149 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:33.705030 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:33.705545 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:33.705685 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:33.707008 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:33.707014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:33.707019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:33.707046 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:33.707086 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:33.707098 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:33.707199 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:33.707421 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 02:51:33.707511 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":243,"threads_started":1}
I20260621 02:51:33.708776 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34927
--builtin_ntp_servers=127.11.109.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 02:51:33.708815 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:33.698406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34927 (local address 127.11.109.1:57475)
0621 02:51:33.699261 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:33.699299 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:33.700869 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:33.701627 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:33.701634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:33.701720 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:33.702253 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:33.702268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:33.703306 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:33.703316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:33.704861 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:33.704874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:33.705774 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:33.705783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:33.706017 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:33.706717 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:51:33.706720 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:33.706755 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:51:33.706827 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:33.707197 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:33.707209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:33.707219 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:51:33.707401 (+ 182us) client_negotiation.cc:240] Negotiation successful
0621 02:51:33.707758 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":261,"threads_started":1}
I20260621 02:51:33.710283 21223 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34927
I20260621 02:51:33.710593 21223 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:33.711436 21223 heartbeater.cc:511] Master 127.11.109.62:34927 requested a full tablet report, sending...
I20260621 02:51:33.713487 21037 ts_manager.cc:194] Registered new tserver with Master: 55f6750dc3924b7fb8a685ca7477aa4a (127.11.109.1:46547)
I20260621 02:51:33.714952 21037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.1:57475
W20260621 02:51:34.026947 21227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:34.027194 21227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:34.027323 21227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:34.034448 21227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:34.034540 21227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:34.034708 21227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:51:34.043447 21227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:34.044617 21227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:34.046314 21227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:34.058725 21235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:34.058805 21233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:34.058903 21232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:34.070451 21227 server_base.cc:1034] running on GCE node
I20260621 02:51:34.071054 21227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:34.071972 21227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:34.073271 21227 hybrid_clock.cc:648] HybridClock initialized: now 1782010294073196 us; error 58 us; skew 500 ppm
I20260621 02:51:34.073585 21227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:34.075899 21227 webserver.cc:460] Webserver started at http://127.11.109.2:39705/ using document root <none> and password file <none>
I20260621 02:51:34.076586 21227 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:34.076715 21227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:34.077036 21227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:34.079881 21227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "17d99d0d89ad4e9fbae37d3a8a97f3a7"
format_stamp: "Formatted at 2026-06-21 02:51:34 on dist-test-slave-bc52"
server_key: "699dc3ee0187f1102f65f1df411b59e2"
server_key_iv: "403a9a1d0ebd806248ff4232cf6f98c6"
server_key_version: "encryptionkey@0"
I20260621 02:51:34.080641 21227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "17d99d0d89ad4e9fbae37d3a8a97f3a7"
format_stamp: "Formatted at 2026-06-21 02:51:34 on dist-test-slave-bc52"
server_key: "699dc3ee0187f1102f65f1df411b59e2"
server_key_iv: "403a9a1d0ebd806248ff4232cf6f98c6"
server_key_version: "encryptionkey@0"
I20260621 02:51:34.085913 21227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:51:34.089257 21241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:34.090582 21227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:51:34.090764 21227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "17d99d0d89ad4e9fbae37d3a8a97f3a7"
format_stamp: "Formatted at 2026-06-21 02:51:34 on dist-test-slave-bc52"
server_key: "699dc3ee0187f1102f65f1df411b59e2"
server_key_iv: "403a9a1d0ebd806248ff4232cf6f98c6"
server_key_version: "encryptionkey@0"
I20260621 02:51:34.091024 21227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:34.107400 21227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:34.108233 21227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:34.108539 21227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:34.109517 21227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:34.110893 21227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:34.111016 21227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:34.111151 21227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:34.111240 21227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:34.143873 21227 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:40103
I20260621 02:51:34.143959 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:40103 every 8 connection(s)
I20260621 02:51:34.145164 21227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:51:34.150563 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21227
I20260621 02:51:34.150738 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:51:34.151182 11700 external_mini_cluster.cc:1428] Setting key 43b7e9c42baddb3a054fdbf56b3173c8
I20260621 02:51:34.154050 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34927
--builtin_ntp_servers=127.11.109.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 02:51:34.160624 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.147707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:43579 (local address 127.11.109.62:34927)
0621 02:51:34.147925 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:34.147932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:34.149490 (+ 1558us) server_negotiation.cc:407] Connection header received
0621 02:51:34.152118 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:34.152122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:34.152187 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:34.152316 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:34.153650 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.154764 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.157358 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.157999 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.158131 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.159791 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:34.159798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:34.159803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:34.159824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:34.159862 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:34.159877 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:34.159963 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:34.160219 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 02:51:34.160315 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:51:34.161873 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.148274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34927 (local address 127.11.109.2:43579)
0621 02:51:34.149315 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:34.149363 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:34.151629 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:34.152554 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:34.152562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:34.152654 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:34.153461 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:34.153489 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.154930 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.154939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:34.157207 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:34.157224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.158236 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.158245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:34.158574 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.159418 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:51:34.159424 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:34.159478 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:51:34.159579 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:34.159978 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:34.159990 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:34.160001 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:51:34.160174 (+ 173us) client_negotiation.cc:240] Negotiation successful
0621 02:51:34.160599 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":299,"threads_started":1}
I20260621 02:51:34.163187 21354 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34927
I20260621 02:51:34.163494 21354 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:34.164230 21354 heartbeater.cc:511] Master 127.11.109.62:34927 requested a full tablet report, sending...
I20260621 02:51:34.165566 21037 ts_manager.cc:194] Registered new tserver with Master: 17d99d0d89ad4e9fbae37d3a8a97f3a7 (127.11.109.2:40103)
I20260621 02:51:34.166365 21037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.2:43579
W20260621 02:51:34.471398 21357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:34.471745 21357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:34.471907 21357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:34.478982 21357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:34.479104 21357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:34.479279 21357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:51:34.488263 21357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:34.489676 21357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:34.491324 21357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:34.503892 21362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:34.503960 21363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:34.503993 21365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:34.504909 21357 server_base.cc:1034] running on GCE node
I20260621 02:51:34.505535 21357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:34.506474 21357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:34.507671 21357 hybrid_clock.cc:648] HybridClock initialized: now 1782010294507625 us; error 34 us; skew 500 ppm
I20260621 02:51:34.508054 21357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:34.511093 21357 webserver.cc:460] Webserver started at http://127.11.109.3:41497/ using document root <none> and password file <none>
I20260621 02:51:34.511776 21357 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:34.511891 21357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:34.512216 21357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:34.514959 21357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "dfec4adb65f94b669d543976e4df677c"
format_stamp: "Formatted at 2026-06-21 02:51:34 on dist-test-slave-bc52"
server_key: "0f1a252c8cf342a4ae695d1c7a00238c"
server_key_iv: "721018f84a9f27535782cfc652245bec"
server_key_version: "encryptionkey@0"
I20260621 02:51:34.515759 21357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "dfec4adb65f94b669d543976e4df677c"
format_stamp: "Formatted at 2026-06-21 02:51:34 on dist-test-slave-bc52"
server_key: "0f1a252c8cf342a4ae695d1c7a00238c"
server_key_iv: "721018f84a9f27535782cfc652245bec"
server_key_version: "encryptionkey@0"
I20260621 02:51:34.520742 21357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:51:34.523968 21371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:34.525153 21357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:51:34.525383 21357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "dfec4adb65f94b669d543976e4df677c"
format_stamp: "Formatted at 2026-06-21 02:51:34 on dist-test-slave-bc52"
server_key: "0f1a252c8cf342a4ae695d1c7a00238c"
server_key_iv: "721018f84a9f27535782cfc652245bec"
server_key_version: "encryptionkey@0"
I20260621 02:51:34.525586 21357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:34.555491 21357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:34.556351 21357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:34.556682 21357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:34.557643 21357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:34.558995 21357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:34.559120 21357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:34.559242 21357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:34.559325 21357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:34.590854 21357 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:33393
I20260621 02:51:34.590935 21483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:33393 every 8 connection(s)
I20260621 02:51:34.592115 21357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:51:34.597963 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21357
I20260621 02:51:34.598141 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:51:34.598562 11700 external_mini_cluster.cc:1428] Setting key 25300f06a6d9688e84437736502a09a6
I20260621 02:51:34.604347 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.594661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:37157 (local address 127.11.109.62:34927)
0621 02:51:34.594828 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:34.594832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:34.596290 (+ 1458us) server_negotiation.cc:407] Connection header received
0621 02:51:34.598060 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:34.598065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:34.598160 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:34.598267 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:34.599427 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.600317 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.601863 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.602265 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.602396 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.603568 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:34.603572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:34.603575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:34.603593 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:34.603625 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:34.603637 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:34.603731 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:34.603939 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 02:51:34.604024 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:51:34.605218 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.595274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34927 (local address 127.11.109.3:37157)
0621 02:51:34.596143 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:34.596180 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:34.597811 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:34.598512 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:34.598520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:34.598607 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:34.599277 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:34.599296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.600443 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.600448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:34.601728 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:34.601740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.602470 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.602479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:34.602697 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.603281 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:51:34.603283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:34.603318 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:51:34.603383 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:34.603730 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:34.603741 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:34.603750 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:51:34.603916 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 02:51:34.604248 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":258,"threads_started":1}
I20260621 02:51:34.606441 21484 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34927
I20260621 02:51:34.606719 21484 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:34.607414 21484 heartbeater.cc:511] Master 127.11.109.62:34927 requested a full tablet report, sending...
I20260621 02:51:34.608628 21037 ts_manager.cc:194] Registered new tserver with Master: dfec4adb65f94b669d543976e4df677c (127.11.109.3:33393)
I20260621 02:51:34.609397 21037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.3:37157
I20260621 02:51:34.613281 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 02:51:34.618947 21225 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:34.617386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.11.109.62:34927)
0621 02:51:34.617584 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:34.617592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:34.617981 (+ 389us) server_negotiation.cc:407] Connection header received
0621 02:51:34.618173 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:34.618201 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:34.618297 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:34.618408 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:34.618759 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38742: BlockingRecv error: recv got EOF from 127.0.0.1:38742 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:51:34.619514 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21096
I20260621 02:51:34.626444 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.620377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.11.109.1:46547)
0621 02:51:34.620938 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:34.620945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:34.620967 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:51:34.621016 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:34.621021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:34.621261 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:34.621375 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:34.621689 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.622610 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.623759 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.624251 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.624342 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.625276 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:34.625285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:34.625296 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:34.625831 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:34.625870 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:34.625888 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:34.625983 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:34.626093 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 02:51:34.626184 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":301,"threads_started":1}
I20260621 02:51:34.627483 21218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:34.707437 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21096
I20260621 02:51:34.724303 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21227
I20260621 02:51:34.731519 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.725101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.11.109.2:40103)
0621 02:51:34.725686 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:34.725693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:34.725720 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:51:34.725781 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:34.725787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:34.726007 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:34.726143 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:34.726468 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.727431 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.728548 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.729075 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.729368 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.730252 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:34.730260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:34.730275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:34.730847 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:34.730892 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:34.730910 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:34.731003 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:34.731117 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 02:51:34.731218 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":300,"threads_started":1}
I20260621 02:51:34.733099 21349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:34.813704 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21227
I20260621 02:51:34.829873 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21357
I20260621 02:51:34.837483 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:34.830891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.11.109.3:33393)
0621 02:51:34.831474 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:34.831481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:34.831510 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:51:34.831568 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:34.831574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:34.831780 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:34.831905 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:34.832302 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.833373 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.834563 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:34.835089 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:34.835184 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:34.836125 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:34.836133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:34.836145 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:34.836729 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:34.836775 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:34.836804 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:34.836890 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:34.837014 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 02:51:34.837119 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":296,"threads_started":1}
I20260621 02:51:34.838668 21479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:34.919515 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21357
I20260621 02:51:34.935379 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21006
I20260621 02:51:34.936671 21067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:35.036166 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21006
2026-06-21T02:51:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2356 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T02:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:35Z Disabled control of system clock
I20260621 02:51:35.103745 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44203
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 02:51:35.425452 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:35.425742 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:35.425835 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:35.433815 21503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:35.433921 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:35.433961 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:35.434022 21503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:35.434069 21503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:35.444008 21503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:44203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:35.445266 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:35.447048 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:35.459398 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:35.459328 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:35.459602 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:35.460479 21503 server_base.cc:1034] running on GCE node
I20260621 02:51:35.461325 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:35.462715 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:35.463965 21503 hybrid_clock.cc:648] HybridClock initialized: now 1782010295463922 us; error 35 us; skew 500 ppm
I20260621 02:51:35.464321 21503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:35.466558 21503 webserver.cc:460] Webserver started at http://127.11.109.62:36111/ using document root <none> and password file <none>
I20260621 02:51:35.467254 21503 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:35.467415 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:35.467752 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:35.470830 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "709d91315eed4ec1bd9808220b004af7"
format_stamp: "Formatted at 2026-06-21 02:51:35 on dist-test-slave-bc52"
server_key: "6679f8a8e101e99bcdf79f52abf2d452"
server_key_iv: "2b80716a2d2c0777b09f94ea6fef3e97"
server_key_version: "encryptionkey@0"
I20260621 02:51:35.471678 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "709d91315eed4ec1bd9808220b004af7"
format_stamp: "Formatted at 2026-06-21 02:51:35 on dist-test-slave-bc52"
server_key: "6679f8a8e101e99bcdf79f52abf2d452"
server_key_iv: "2b80716a2d2c0777b09f94ea6fef3e97"
server_key_version: "encryptionkey@0"
I20260621 02:51:35.476708 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:51:35.480185 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:35.481491 21503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:51:35.481685 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "709d91315eed4ec1bd9808220b004af7"
format_stamp: "Formatted at 2026-06-21 02:51:35 on dist-test-slave-bc52"
server_key: "6679f8a8e101e99bcdf79f52abf2d452"
server_key_iv: "2b80716a2d2c0777b09f94ea6fef3e97"
server_key_version: "encryptionkey@0"
I20260621 02:51:35.481907 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:35.487429 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:35.488145 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:35.488453 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:35.506008 21503 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:44203
I20260621 02:51:35.506045 21568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:44203 every 8 connection(s)
I20260621 02:51:35.507491 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:51:35.511013 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:35.513842 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21503
I20260621 02:51:35.514047 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:51:35.514770 11700 external_mini_cluster.cc:1428] Setting key 4c53d282cb2bc3b1e7ddb57881d8fe78
I20260621 02:51:35.523394 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7: Bootstrap starting.
I20260621 02:51:35.527259 21569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:35.528141 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:35.517032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.11.109.62:44203)
0621 02:51:35.521987 (+ 4955us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:35.522004 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:35.522056 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:51:35.524282 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:35.524308 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:35.524804 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:35.525291 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:35.526279 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:35.526289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:35.526307 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:35.526327 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:35.526396 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:35.526418 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:35.526587 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:35.526703 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 02:51:35.526953 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4401,"thread_start_us":4120,"threads_started":1}
I20260621 02:51:35.528512 21569 log.cc:824] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:35.531512 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7: No bootstrap required, opened a new log
I20260621 02:51:35.538436 21569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709d91315eed4ec1bd9808220b004af7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44203 } }
I20260621 02:51:35.538796 21569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:35.538903 21569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709d91315eed4ec1bd9808220b004af7, State: Initialized, Role: FOLLOWER
I20260621 02:51:35.539423 21569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709d91315eed4ec1bd9808220b004af7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44203 } }
I20260621 02:51:35.539652 21569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:51:35.539788 21569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:51:35.539956 21569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:35.542127 21569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709d91315eed4ec1bd9808220b004af7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44203 } }
I20260621 02:51:35.542575 21569 leader_election.cc:302] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709d91315eed4ec1bd9808220b004af7; no voters:
I20260621 02:51:35.543100 21569 leader_election.cc:288] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:51:35.543241 21574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:35.543910 21574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [term 1 LEADER]: Becoming Leader. State: Replica: 709d91315eed4ec1bd9808220b004af7, State: Running, Role: LEADER
I20260621 02:51:35.544548 21574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709d91315eed4ec1bd9808220b004af7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44203 } }
I20260621 02:51:35.545075 21569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:51:35.547160 21576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709d91315eed4ec1bd9808220b004af7. Latest consensus state: current_term: 1 leader_uuid: "709d91315eed4ec1bd9808220b004af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709d91315eed4ec1bd9808220b004af7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44203 } } }
I20260621 02:51:35.547762 21576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:35.547761 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709d91315eed4ec1bd9808220b004af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709d91315eed4ec1bd9808220b004af7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44203 } } }
I20260621 02:51:35.548094 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:35.550202 21582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:35.555105 21582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:35.563017 21582 catalog_manager.cc:1347] Generated new cluster ID: 43cd3579b7324a46a0d148a03ccaaf36
I20260621 02:51:35.563117 21582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:35.613075 21582 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:51:35.613533 21582 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:35.619477 21582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709d91315eed4ec1bd9808220b004af7: Generated new TSK 0
I20260621 02:51:35.620234 21582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:51:35.636392 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44203
--builtin_ntp_servers=127.11.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 02:51:35.956455 21593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:35.956709 21593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:35.956859 21593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:35.964085 21593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:35.964179 21593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:35.964339 21593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:51:35.973333 21593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:35.974550 21593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:35.976305 21593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:35.989100 21599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:35.989188 21598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:35.989209 21601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:35.990259 21593 server_base.cc:1034] running on GCE node
I20260621 02:51:35.990814 21593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:35.991757 21593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:35.992970 21593 hybrid_clock.cc:648] HybridClock initialized: now 1782010295992918 us; error 48 us; skew 500 ppm
I20260621 02:51:35.993364 21593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:35.995429 21593 webserver.cc:460] Webserver started at http://127.11.109.1:38715/ using document root <none> and password file <none>
I20260621 02:51:35.996090 21593 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:35.996214 21593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:35.996518 21593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:35.999297 21593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "188b34f7ac584765bf9d94d2aa274932"
format_stamp: "Formatted at 2026-06-21 02:51:35 on dist-test-slave-bc52"
server_key: "9c34a0d48d4ea1dae8246e7165d7afcb"
server_key_iv: "68895323c752f7117ced27edbaa14d39"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.000057 21593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "188b34f7ac584765bf9d94d2aa274932"
format_stamp: "Formatted at 2026-06-21 02:51:35 on dist-test-slave-bc52"
server_key: "9c34a0d48d4ea1dae8246e7165d7afcb"
server_key_iv: "68895323c752f7117ced27edbaa14d39"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.004840 21593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260621 02:51:36.008136 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.009438 21593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:51:36.009613 21593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "188b34f7ac584765bf9d94d2aa274932"
format_stamp: "Formatted at 2026-06-21 02:51:35 on dist-test-slave-bc52"
server_key: "9c34a0d48d4ea1dae8246e7165d7afcb"
server_key_iv: "68895323c752f7117ced27edbaa14d39"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.009836 21593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:36.015179 21593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:36.015950 21593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:36.016245 21593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:36.017081 21593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:36.018430 21593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:36.018548 21593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.018677 21593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:36.018769 21593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.052174 21593 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:36379
I20260621 02:51:36.052259 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:36379 every 8 connection(s)
I20260621 02:51:36.053525 21593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:51:36.057081 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21593
I20260621 02:51:36.057317 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:51:36.057978 11700 external_mini_cluster.cc:1428] Setting key b61e8afea7648bf0c20e445b4ffd85e1
I20260621 02:51:36.060956 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44203
--builtin_ntp_servers=127.11.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 02:51:36.062288 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.056047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:37589 (local address 127.11.109.62:44203)
0621 02:51:36.056445 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:36.056449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:36.057843 (+ 1394us) server_negotiation.cc:407] Connection header received
0621 02:51:36.059863 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:36.059867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:36.059935 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:36.060057 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:36.061427 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:36.061436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:36.061444 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:36.061466 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:36.061519 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:36.061538 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:36.061676 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:36.061842 (+ 166us) server_negotiation.cc:299] Negotiation successful
0621 02:51:36.061905 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":212,"threads_started":1}
I20260621 02:51:36.062954 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.056622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44203 (local address 127.11.109.1:37589)
0621 02:51:36.057669 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:36.057724 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:36.059544 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:36.060284 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:36.060292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:36.060379 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:36.061053 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:51:36.061055 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:36.061113 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:51:36.061264 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:36.061654 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:36.061661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:36.061666 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:51:36.061866 (+ 200us) client_negotiation.cc:240] Negotiation successful
0621 02:51:36.062042 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":319,"threads_started":1}
I20260621 02:51:36.064371 21720 heartbeater.cc:347] Connected to a master server at 127.11.109.62:44203
I20260621 02:51:36.064654 21720 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:36.065272 21720 heartbeater.cc:511] Master 127.11.109.62:44203 requested a full tablet report, sending...
I20260621 02:51:36.067180 21534 ts_manager.cc:194] Registered new tserver with Master: 188b34f7ac584765bf9d94d2aa274932 (127.11.109.1:36379)
W20260621 02:51:36.375125 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:36.375432 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:36.375598 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:36.382692 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:36.382828 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:36.383004 21724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:51:36.391798 21724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:36.393039 21724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:36.394732 21724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:36.407001 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:36.406945 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:36.407056 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:36.407872 21724 server_base.cc:1034] running on GCE node
I20260621 02:51:36.408417 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:36.409312 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:36.410526 21724 hybrid_clock.cc:648] HybridClock initialized: now 1782010296410482 us; error 54 us; skew 500 ppm
I20260621 02:51:36.410832 21724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:36.413033 21724 webserver.cc:460] Webserver started at http://127.11.109.2:38047/ using document root <none> and password file <none>
I20260621 02:51:36.413786 21724 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:36.413910 21724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:36.414206 21724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:36.416895 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "39c1e896b67140438e490ef2b04ed07f"
format_stamp: "Formatted at 2026-06-21 02:51:36 on dist-test-slave-bc52"
server_key: "60d9e5f0ceb591219022e2e81c1704ad"
server_key_iv: "f0e5e2b4226012e7dcbb2053385fa05d"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.417706 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "39c1e896b67140438e490ef2b04ed07f"
format_stamp: "Formatted at 2026-06-21 02:51:36 on dist-test-slave-bc52"
server_key: "60d9e5f0ceb591219022e2e81c1704ad"
server_key_iv: "f0e5e2b4226012e7dcbb2053385fa05d"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.422418 21724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 02:51:36.425520 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.426741 21724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:51:36.426918 21724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "39c1e896b67140438e490ef2b04ed07f"
format_stamp: "Formatted at 2026-06-21 02:51:36 on dist-test-slave-bc52"
server_key: "60d9e5f0ceb591219022e2e81c1704ad"
server_key_iv: "f0e5e2b4226012e7dcbb2053385fa05d"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.427134 21724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:36.432684 21724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:36.433506 21724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:36.433766 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:36.434778 21724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:36.436475 21724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:36.436591 21724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.436694 21724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:36.436796 21724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.468973 21724 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:34459
I20260621 02:51:36.469020 21850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:34459 every 8 connection(s)
I20260621 02:51:36.470278 21724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:51:36.471482 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21724
I20260621 02:51:36.471701 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:51:36.472304 11700 external_mini_cluster.cc:1428] Setting key 4af3cfdae49fbb0bba08c8c2363d2e87
I20260621 02:51:36.475322 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44203
--builtin_ntp_servers=127.11.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 02:51:36.478942 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.473133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:58411 (local address 127.11.109.62:44203)
0621 02:51:36.473363 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:36.473368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:36.474888 (+ 1520us) server_negotiation.cc:407] Connection header received
0621 02:51:36.476651 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:36.476656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:36.476736 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:36.476855 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:36.478241 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:36.478245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:36.478248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:36.478263 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:36.478293 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:36.478304 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:36.478434 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:36.478619 (+ 185us) server_negotiation.cc:299] Negotiation successful
0621 02:51:36.478685 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260621 02:51:36.479888 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.473695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44203 (local address 127.11.109.2:58411)
0621 02:51:36.474744 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:36.474779 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:36.476389 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:36.477087 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:36.477096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:36.477236 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:36.477934 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:51:36.477936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:36.477976 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:51:36.478060 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:36.478398 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:36.478410 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:36.478420 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:51:36.478707 (+ 287us) client_negotiation.cc:240] Negotiation successful
0621 02:51:36.478962 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":345,"threads_started":1}
I20260621 02:51:36.481230 21851 heartbeater.cc:347] Connected to a master server at 127.11.109.62:44203
I20260621 02:51:36.481518 21851 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:36.482082 21851 heartbeater.cc:511] Master 127.11.109.62:44203 requested a full tablet report, sending...
I20260621 02:51:36.483458 21534 ts_manager.cc:194] Registered new tserver with Master: 39c1e896b67140438e490ef2b04ed07f (127.11.109.2:34459)
W20260621 02:51:36.792851 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:36.793095 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:36.793283 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:36.800432 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:36.800529 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:36.800700 21854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:51:36.809454 21854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:36.810732 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:36.812414 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:36.824971 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:36.825064 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:36.825037 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:36.830557 21854 server_base.cc:1034] running on GCE node
I20260621 02:51:36.831254 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:36.832185 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:36.833411 21854 hybrid_clock.cc:648] HybridClock initialized: now 1782010296833354 us; error 47 us; skew 500 ppm
I20260621 02:51:36.833748 21854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:51:36.835961 21854 webserver.cc:460] Webserver started at http://127.11.109.3:37007/ using document root <none> and password file <none>
I20260621 02:51:36.836624 21854 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:36.836766 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:36.837083 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:36.839913 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "d40094738227456682887389cbf9f696"
format_stamp: "Formatted at 2026-06-21 02:51:36 on dist-test-slave-bc52"
server_key: "5ab55877984f7ab97f437c20f1aa629b"
server_key_iv: "a3f4956114f02b4e6810fc6ddbbf8bef"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.840696 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "d40094738227456682887389cbf9f696"
format_stamp: "Formatted at 2026-06-21 02:51:36 on dist-test-slave-bc52"
server_key: "5ab55877984f7ab97f437c20f1aa629b"
server_key_iv: "a3f4956114f02b4e6810fc6ddbbf8bef"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.845613 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:51:36.848876 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.850384 21854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:51:36.850541 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "d40094738227456682887389cbf9f696"
format_stamp: "Formatted at 2026-06-21 02:51:36 on dist-test-slave-bc52"
server_key: "5ab55877984f7ab97f437c20f1aa629b"
server_key_iv: "a3f4956114f02b4e6810fc6ddbbf8bef"
server_key_version: "encryptionkey@0"
I20260621 02:51:36.850833 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:36.856271 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:36.857008 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:36.857327 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:36.858184 21854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:36.859467 21854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:36.859598 21854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.859702 21854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:36.859773 21854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:36.892264 21854 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:33179
I20260621 02:51:36.892315 21980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:33179 every 8 connection(s)
I20260621 02:51:36.893549 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:51:36.896890 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 21854
I20260621 02:51:36.897078 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:51:36.897526 11700 external_mini_cluster.cc:1428] Setting key 709f725db26550935569560adb8048b1
I20260621 02:51:36.901526 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.895974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:59583 (local address 127.11.109.62:44203)
0621 02:51:36.896185 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:36.896191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:36.897793 (+ 1602us) server_negotiation.cc:407] Connection header received
0621 02:51:36.899397 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:36.899402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:36.899463 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:36.899550 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:36.900922 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:36.900926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:36.900929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:36.900943 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:36.900971 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:36.900982 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:36.901072 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:36.901288 (+ 216us) server_negotiation.cc:299] Negotiation successful
0621 02:51:36.901334 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:51:36.902407 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.896579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44203 (local address 127.11.109.3:59583)
0621 02:51:36.897615 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:36.897667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:36.899163 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:36.899817 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:36.899828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:36.899947 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:36.900661 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:51:36.900663 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:36.900698 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:51:36.900777 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:36.901065 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:36.901072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:36.901077 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:51:36.901345 (+ 268us) client_negotiation.cc:240] Negotiation successful
0621 02:51:36.901520 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":321,"threads_started":1}
I20260621 02:51:36.903578 21981 heartbeater.cc:347] Connected to a master server at 127.11.109.62:44203
I20260621 02:51:36.903864 21981 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:36.904395 21981 heartbeater.cc:511] Master 127.11.109.62:44203 requested a full tablet report, sending...
I20260621 02:51:36.905696 21534 ts_manager.cc:194] Registered new tserver with Master: d40094738227456682887389cbf9f696 (127.11.109.3:33179)
I20260621 02:51:36.912387 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 02:51:36.918231 21722 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:51:36.916754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.11.109.62:44203)
0621 02:51:36.916924 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:36.916930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:36.917482 (+ 552us) server_negotiation.cc:407] Connection header received
0621 02:51:36.917595 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:36.917599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:36.917644 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:36.917718 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:36.918009 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59294: BlockingRecv error: recv got EOF from 127.0.0.1:59294 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:51:36.918931 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21593
I20260621 02:51:36.922399 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:36.919918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.11.109.1:36379)
0621 02:51:36.920413 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:36.920419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:36.920442 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:51:36.920493 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:36.920498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:36.920669 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:36.920788 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:36.921726 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:36.921733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:36.921743 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:36.921768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:36.921811 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:36.921826 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:36.921962 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:36.922060 (+ 98us) server_negotiation.cc:299] Negotiation successful
0621 02:51:36.922154 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":235,"threads_started":1}
I20260621 02:51:36.923512 21715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:37.003294 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21593
I20260621 02:51:37.019471 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21724
I20260621 02:51:37.023164 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:37.020425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.11.109.2:34459)
0621 02:51:37.020896 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:37.020903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:37.020928 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:51:37.020994 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:37.021000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:37.021310 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:37.021492 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:37.022422 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:37.022428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:37.022440 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:37.022457 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:37.022498 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:37.022512 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:37.022615 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:37.022749 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 02:51:37.022868 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260621 02:51:37.024430 21846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:37.105022 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21724
I20260621 02:51:37.120605 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21854
I20260621 02:51:37.124301 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:37.121624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.11.109.3:33179)
0621 02:51:37.122249 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:37.122256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:37.122285 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:51:37.122344 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:37.122350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:37.122562 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:37.122713 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:37.123571 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:37.123583 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:37.123596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:51:37.123611 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:37.123656 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:51:37.123668 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:37.123801 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:37.123882 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 02:51:37.123990 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":321,"threads_started":1}
I20260621 02:51:37.125665 21976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:37.206142 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21854
I20260621 02:51:37.221323 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 21503
I20260621 02:51:37.222492 21564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:37.322499 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 21503
2026-06-21T02:51:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2286 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:51:37 dist-test-slave-bc52 krb5kdc[21999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:51:37 dist-test-slave-bc52 krb5kdc[21999](info): set up 2 sockets
Jun 21 02:51:37 dist-test-slave-bc52 krb5kdc[21999](info): commencing operation
krb5kdc: starting...
W20260621 02:51:38.426141 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:51:38 dist-test-slave-bc52 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:51:38.640271 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37505
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:38.954233 22012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:38.954548 22012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:38.954731 22012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:38.961699 22012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:51:38.961850 22012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:38.961951 22012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:38.962046 22012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:51:38.962126 22012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:51:38.971710 22012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:38.972963 22012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:38.974251 22012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:38.985543 22018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:38.985703 22017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:38.985706 22020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:38.986900 22012 server_base.cc:1034] running on GCE node
I20260621 02:51:38.987681 22012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:38.988971 22012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:38.990211 22012 hybrid_clock.cc:648] HybridClock initialized: now 1782010298990134 us; error 71 us; skew 500 ppm
I20260621 02:51:38.990545 22012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:38 dist-test-slave-bc52 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:38.997061 22012 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:51:38.998489 22012 webserver.cc:460] Webserver started at http://127.11.109.62:41811/ using document root <none> and password file <none>
I20260621 02:51:38.999163 22012 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:38.999315 22012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:38.999615 22012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:39.002573 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "405f0d2ddc14440a8e4f6fcb86028563"
format_stamp: "Formatted at 2026-06-21 02:51:38 on dist-test-slave-bc52"
server_key: "628f624d59b3d48ca73ff60186399a48"
server_key_iv: "71b4d6aba2af33ce09e8d220b7659847"
server_key_version: "encryptionkey@0"
I20260621 02:51:39.003325 22012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "405f0d2ddc14440a8e4f6fcb86028563"
format_stamp: "Formatted at 2026-06-21 02:51:38 on dist-test-slave-bc52"
server_key: "628f624d59b3d48ca73ff60186399a48"
server_key_iv: "71b4d6aba2af33ce09e8d220b7659847"
server_key_version: "encryptionkey@0"
I20260621 02:51:39.008296 22012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:51:39.011551 22027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:39.012802 22012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:51:39.012981 22012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "405f0d2ddc14440a8e4f6fcb86028563"
format_stamp: "Formatted at 2026-06-21 02:51:38 on dist-test-slave-bc52"
server_key: "628f624d59b3d48ca73ff60186399a48"
server_key_iv: "71b4d6aba2af33ce09e8d220b7659847"
server_key_version: "encryptionkey@0"
I20260621 02:51:39.013167 22012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:39.033978 22012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:39.037607 22012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:39.037901 22012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:39.054836 22012 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:37505
I20260621 02:51:39.054869 22078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:37505 every 8 connection(s)
I20260621 02:51:39.056056 22012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:51:39.059481 22079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:39.059751 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 22012
I20260621 02:51:39.059978 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:51:39.060608 11700 external_mini_cluster.cc:1428] Setting key 48a548677399fea68d15dc2bac13b062
I20260621 02:51:39.073333 22079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563: Bootstrap starting.
Jun 21 02:51:39 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:39.076200 22079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:39.077524 22079 log.cc:824] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:39.080727 22079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563: No bootstrap required, opened a new log
I20260621 02:51:39.084216 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:39.062613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.11.109.62:37505)
0621 02:51:39.063409 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:39.063424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:39.063471 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:51:39.066071 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:39.066098 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:39.066566 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:39.066991 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:39.068269 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:39.070240 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:39.071503 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:39.072487 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:39.072660 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:39.076392 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:39.076404 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:39.076421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:39.076461 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:39.080243 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:39.080962 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:39.080968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:39.080973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:39.081218 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:39.081600 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:39.081607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:39.081610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:39.082099 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:39.082435 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:39.082865 (+ 430us) server_negotiation.cc:299] Negotiation successful
0621 02:51:39.083250 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":302,"threads_started":1}
I20260621 02:51:39.088934 22079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405f0d2ddc14440a8e4f6fcb86028563" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37505 } }
I20260621 02:51:39.089417 22079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:39.089538 22079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 405f0d2ddc14440a8e4f6fcb86028563, State: Initialized, Role: FOLLOWER
I20260621 02:51:39.090173 22079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405f0d2ddc14440a8e4f6fcb86028563" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37505 } }
I20260621 02:51:39.090435 22079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:51:39.090556 22079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:51:39.090734 22079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:39.093236 22079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405f0d2ddc14440a8e4f6fcb86028563" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37505 } }
I20260621 02:51:39.093743 22079 leader_election.cc:302] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 405f0d2ddc14440a8e4f6fcb86028563; no voters:
I20260621 02:51:39.094293 22079 leader_election.cc:288] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:51:39.094419 22084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:39.095125 22084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [term 1 LEADER]: Becoming Leader. State: Replica: 405f0d2ddc14440a8e4f6fcb86028563, State: Running, Role: LEADER
I20260621 02:51:39.095742 22084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405f0d2ddc14440a8e4f6fcb86028563" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37505 } }
I20260621 02:51:39.096329 22079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:51:39.098680 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 405f0d2ddc14440a8e4f6fcb86028563. Latest consensus state: current_term: 1 leader_uuid: "405f0d2ddc14440a8e4f6fcb86028563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405f0d2ddc14440a8e4f6fcb86028563" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37505 } } }
I20260621 02:51:39.099085 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:39.099409 22085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "405f0d2ddc14440a8e4f6fcb86028563" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405f0d2ddc14440a8e4f6fcb86028563" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37505 } } }
I20260621 02:51:39.099845 22085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563 [sys.catalog]: This master's current role is: LEADER
I20260621 02:51:39.101137 22089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:51:39.106055 22089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:51:39.114667 22089 catalog_manager.cc:1347] Generated new cluster ID: 8a504e057080484eab838998c52788f5
I20260621 02:51:39.114789 22089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:51:39.140221 22089 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:51:39.141537 22089 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:51:39.148370 22089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563: Generated new TSK 0
I20260621 02:51:39.149318 22089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:51:39.216626 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37505
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:39.531402 22105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:39.531709 22105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:39.531878 22105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:39.538834 22105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:39.538972 22105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:39.539155 22105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:51:39.548537 22105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:39.549803 22105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:39.551083 22105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:39.563517 22111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:39.563577 22110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:39.563795 22113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:39.564368 22105 server_base.cc:1034] running on GCE node
I20260621 02:51:39.564968 22105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:39.565896 22105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:39.567149 22105 hybrid_clock.cc:648] HybridClock initialized: now 1782010299567118 us; error 59 us; skew 500 ppm
I20260621 02:51:39.567456 22105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:39 dist-test-slave-bc52 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010299, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:39.570441 22105 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:51:39.571838 22105 webserver.cc:460] Webserver started at http://127.11.109.1:40631/ using document root <none> and password file <none>
I20260621 02:51:39.572504 22105 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:39.572642 22105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:39.572919 22105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:39.575685 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "cfde5de5891c43959e2df726c5561198"
format_stamp: "Formatted at 2026-06-21 02:51:39 on dist-test-slave-bc52"
server_key: "bca14ef908297d0c3f238b3ed7812512"
server_key_iv: "39f1297d4937e3545c0be8470334e123"
server_key_version: "encryptionkey@0"
I20260621 02:51:39.576397 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "cfde5de5891c43959e2df726c5561198"
format_stamp: "Formatted at 2026-06-21 02:51:39 on dist-test-slave-bc52"
server_key: "bca14ef908297d0c3f238b3ed7812512"
server_key_iv: "39f1297d4937e3545c0be8470334e123"
server_key_version: "encryptionkey@0"
I20260621 02:51:39.581341 22105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:51:39.584689 22120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:39.586141 22105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:51:39.586264 22105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "cfde5de5891c43959e2df726c5561198"
format_stamp: "Formatted at 2026-06-21 02:51:39 on dist-test-slave-bc52"
server_key: "bca14ef908297d0c3f238b3ed7812512"
server_key_iv: "39f1297d4937e3545c0be8470334e123"
server_key_version: "encryptionkey@0"
I20260621 02:51:39.586422 22105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:39.600301 22105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:39.604039 22105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:39.604264 22105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:39.605051 22105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:39.606271 22105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:39.606350 22105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:39.606427 22105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:39.606473 22105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:39.639379 22105 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:41013
I20260621 02:51:39.639472 22232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:41013 every 8 connection(s)
I20260621 02:51:39.640610 22105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:51:39.648291 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 22105
I20260621 02:51:39.648504 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:51:39.649268 11700 external_mini_cluster.cc:1428] Setting key 968b64d3220357261509a114fdab0f38
Jun 21 02:51:39 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010299, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:39.660403 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:39.643171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:41911 (local address 127.11.109.62:37505)
0621 02:51:39.643578 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:39.643583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:39.644561 (+ 978us) server_negotiation.cc:407] Connection header received
0621 02:51:39.646340 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:39.646345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:39.646427 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:39.646538 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:39.648012 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:39.649258 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:39.651092 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:39.651510 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:39.651662 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:39.655028 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:39.655032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:39.655035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:39.655065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:39.657516 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:39.658210 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:39.658219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:39.658222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:39.658347 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:39.658770 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:39.658776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:39.658779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:39.659026 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:39.659186 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:39.659845 (+ 659us) server_negotiation.cc:299] Negotiation successful
0621 02:51:39.660050 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":204,"threads_started":1}
I20260621 02:51:39.661361 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:39.643703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37505 (local address 127.11.109.1:41911)
0621 02:51:39.644393 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:39.644452 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:39.646084 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:39.646764 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:39.646772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:39.647240 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:39.647784 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:39.647802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:39.649423 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:39.649427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:39.650915 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:39.650942 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:39.651741 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:39.651748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:39.652003 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:39.652659 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:39.652698 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:39.654821 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:39.657642 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:39.657652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:39.657685 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:39.658079 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:39.658465 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:39.658470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:39.658474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:39.658658 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:39.659162 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:39.659170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:39.659506 (+ 336us) client_negotiation.cc:769] Sending connection context
0621 02:51:39.659749 (+ 243us) client_negotiation.cc:240] Negotiation successful
0621 02:51:39.660235 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":276,"threads_started":1}
I20260621 02:51:39.662856 22233 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37505
I20260621 02:51:39.663187 22233 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:39.664038 22233 heartbeater.cc:511] Master 127.11.109.62:37505 requested a full tablet report, sending...
I20260621 02:51:39.666070 22044 ts_manager.cc:194] Registered new tserver with Master: cfde5de5891c43959e2df726c5561198 (127.11.109.1:41013)
I20260621 02:51:39.667590 22044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:41911
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:51:39.697911 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37505
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:40.015316 22239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:40.015590 22239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:40.015686 22239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:40.023001 22239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:40.023114 22239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:40.023231 22239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:51:40.032711 22239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:40.033931 22239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:40.035177 22239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:40.047864 22245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:40.047950 22244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:40.047927 22247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:40.049166 22239 server_base.cc:1034] running on GCE node
I20260621 02:51:40.049793 22239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:40.050809 22239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:40.052011 22239 hybrid_clock.cc:648] HybridClock initialized: now 1782010300051963 us; error 62 us; skew 500 ppm
I20260621 02:51:40.052330 22239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:40 dist-test-slave-bc52 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:40.055704 22239 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:51:40.057228 22239 webserver.cc:460] Webserver started at http://127.11.109.2:36047/ using document root <none> and password file <none>
I20260621 02:51:40.057920 22239 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:40.058058 22239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:40.058343 22239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:40.061096 22239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "bff79592c8b047ef85081f738c9d8f2b"
format_stamp: "Formatted at 2026-06-21 02:51:40 on dist-test-slave-bc52"
server_key: "ecbd26b6955751b422358ac33a72a941"
server_key_iv: "639422120cc5b6ee99f7fc8f83421323"
server_key_version: "encryptionkey@0"
I20260621 02:51:40.061946 22239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "bff79592c8b047ef85081f738c9d8f2b"
format_stamp: "Formatted at 2026-06-21 02:51:40 on dist-test-slave-bc52"
server_key: "ecbd26b6955751b422358ac33a72a941"
server_key_iv: "639422120cc5b6ee99f7fc8f83421323"
server_key_version: "encryptionkey@0"
I20260621 02:51:40.066979 22239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:51:40.070331 22254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:40.071583 22239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:51:40.071775 22239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "bff79592c8b047ef85081f738c9d8f2b"
format_stamp: "Formatted at 2026-06-21 02:51:40 on dist-test-slave-bc52"
server_key: "ecbd26b6955751b422358ac33a72a941"
server_key_iv: "639422120cc5b6ee99f7fc8f83421323"
server_key_version: "encryptionkey@0"
I20260621 02:51:40.071978 22239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:40.098626 22239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:40.102420 22239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:40.102710 22239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:40.103621 22239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:40.104897 22239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:40.104997 22239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:40.105135 22239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:40.105237 22239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260621 02:51:40.137445 22239 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:34981
I20260621 02:51:40.137503 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:34981 every 8 connection(s)
I20260621 02:51:40.138721 22239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:51:40.146864 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 22239
I20260621 02:51:40.147079 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:51:40.147605 11700 external_mini_cluster.cc:1428] Setting key c6970c9cbf7d7b9e081fa0e91058836b
Jun 21 02:51:40 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:40.159119 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.141128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:42151 (local address 127.11.109.62:37505)
0621 02:51:40.141347 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.141353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.142559 (+ 1206us) server_negotiation.cc:407] Connection header received
0621 02:51:40.144342 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.144347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.144414 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.144502 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:40.146057 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.146772 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.148907 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.149584 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.149733 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.153702 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:40.153707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:40.153710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:40.153740 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:40.156157 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.156832 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.156836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.156844 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.156963 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.157442 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.157447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.157451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.157712 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:40.157882 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.158523 (+ 641us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.158748 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:51:40.160058 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.141630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37505 (local address 127.11.109.2:42151)
0621 02:51:40.142403 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.142440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.144073 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.144722 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.144730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.145308 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:40.145904 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.145920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.146922 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.146930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.148759 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.148786 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.149846 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.149854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.150130 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.150825 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:40.150857 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:40.153488 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:40.156292 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:40.156301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:40.156316 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:40.156708 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:40.157070 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:40.157076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:40.157079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:40.157328 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:40.157858 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:40.157866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:40.158177 (+ 311us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.158432 (+ 255us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.158914 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":263,"threads_started":1}
I20260621 02:51:40.161454 22367 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37505
I20260621 02:51:40.161835 22367 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:40.162674 22367 heartbeater.cc:511] Master 127.11.109.62:37505 requested a full tablet report, sending...
I20260621 02:51:40.164167 22044 ts_manager.cc:194] Registered new tserver with Master: bff79592c8b047ef85081f738c9d8f2b (127.11.109.2:34981)
I20260621 02:51:40.164955 22044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:42151
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:51:40.198320 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37505
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:51:40.514848 22372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:51:40.515162 22372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:51:40.515339 22372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:51:40.522429 22372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:51:40.522583 22372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:51:40.522756 22372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:51:40.533437 22372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:51:40.534677 22372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:51:40.535979 22372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:51:40.549021 22378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:40.549073 22380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:51:40.549069 22377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:51:40.549873 22372 server_base.cc:1034] running on GCE node
I20260621 02:51:40.550451 22372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:51:40.551369 22372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:51:40.552594 22372 hybrid_clock.cc:648] HybridClock initialized: now 1782010300552538 us; error 51 us; skew 500 ppm
I20260621 02:51:40.552908 22372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:51:40 dist-test-slave-bc52 krb5kdc[21999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:51:40.555960 22372 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:51:40.557384 22372 webserver.cc:460] Webserver started at http://127.11.109.3:33335/ using document root <none> and password file <none>
I20260621 02:51:40.558046 22372 fs_manager.cc:359] Metadata directory not provided
I20260621 02:51:40.558184 22372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:51:40.558473 22372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:51:40.561479 22372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "811894b09fae47e5bf6732c114587b89"
format_stamp: "Formatted at 2026-06-21 02:51:40 on dist-test-slave-bc52"
server_key: "9c659c4b035eb354467b0242f20c7e98"
server_key_iv: "9064180725f030cb99df0f516a54ae75"
server_key_version: "encryptionkey@0"
I20260621 02:51:40.562243 22372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "811894b09fae47e5bf6732c114587b89"
format_stamp: "Formatted at 2026-06-21 02:51:40 on dist-test-slave-bc52"
server_key: "9c659c4b035eb354467b0242f20c7e98"
server_key_iv: "9064180725f030cb99df0f516a54ae75"
server_key_version: "encryptionkey@0"
I20260621 02:51:40.567585 22372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:51:40.571429 22387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:40.572641 22372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:51:40.572813 22372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "811894b09fae47e5bf6732c114587b89"
format_stamp: "Formatted at 2026-06-21 02:51:40 on dist-test-slave-bc52"
server_key: "9c659c4b035eb354467b0242f20c7e98"
server_key_iv: "9064180725f030cb99df0f516a54ae75"
server_key_version: "encryptionkey@0"
I20260621 02:51:40.573005 22372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:51:40.592159 22372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:51:40.595907 22372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:51:40.596201 22372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:51:40.597164 22372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:51:40.598472 22372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:51:40.598570 22372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:40.598718 22372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:51:40.598788 22372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:51:40.631254 22372 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:33871
I20260621 02:51:40.631335 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:33871 every 8 connection(s)
I20260621 02:51:40.632504 22372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:51:40.637444 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 22372
I20260621 02:51:40.637625 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:51:40.638017 11700 external_mini_cluster.cc:1428] Setting key b64fb6612974997e6c512868d82654b2
Jun 21 02:51:40 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010300, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:51:40.652141 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.635004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:57507 (local address 127.11.109.62:37505)
0621 02:51:40.635182 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.635187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.636331 (+ 1144us) server_negotiation.cc:407] Connection header received
0621 02:51:40.638187 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.638193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.638311 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.638426 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:40.640356 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.641518 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.643319 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.643939 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.644060 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.647110 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:40.647114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:40.647117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:40.647148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:40.649511 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.650116 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.650119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.650121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.650236 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.650573 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.650576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.650578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.650843 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:40.650960 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.651584 (+ 624us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.651799 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:51:40.653111 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.635484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37505 (local address 127.11.109.3:57507)
0621 02:51:40.636158 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.636203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.637907 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.638684 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.638696 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.639455 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:51:40.640147 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.640175 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.641687 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.641695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.643174 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.643185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.644139 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.644144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.644320 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.644902 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:51:40.644927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:51:40.646941 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:51:40.649624 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:40.649630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:40.649640 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:40.650020 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:40.650326 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:51:40.650328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:51:40.650331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:51:40.650482 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:51:40.650950 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:51:40.650960 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:51:40.651226 (+ 266us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.651499 (+ 273us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.651967 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260621 02:51:40.654345 22500 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37505
I20260621 02:51:40.654590 22500 heartbeater.cc:464] Registering TS with master...
I20260621 02:51:40.655237 22500 heartbeater.cc:511] Master 127.11.109.62:37505 requested a full tablet report, sending...
I20260621 02:51:40.656430 22044 ts_manager.cc:194] Registered new tserver with Master: 811894b09fae47e5bf6732c114587b89 (127.11.109.3:33871)
I20260621 02:51:40.657269 22044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:57507
I20260621 02:51:40.665328 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:51:40.670981 22233 heartbeater.cc:503] Master 127.11.109.62:37505 was elected leader, sending a full tablet report...
I20260621 02:51:40.681854 22235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.669483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.11.109.62:37505)
0621 02:51:40.669661 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.669667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.670151 (+ 484us) server_negotiation.cc:407] Connection header received
0621 02:51:40.670267 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.670271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.670323 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.670396 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:40.671927 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.672830 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.674039 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.674422 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.674519 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.676810 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:40.676814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:40.676817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:40.676846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:40.679558 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.680159 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.680163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.680165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.680280 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.680603 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.680606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.680608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.680877 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:40.681047 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.681357 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.681494 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:51:40.685359 22044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:51:40.691622 22044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:51:40.725658 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.703400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.11.109.1:41013)
0621 02:51:40.714098 (+ 10698us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.714111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.715163 (+ 1052us) server_negotiation.cc:407] Connection header received
0621 02:51:40.715428 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.715441 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.715735 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.715954 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:40.717226 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.719300 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.723041 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.724516 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.724668 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.724939 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.725140 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.725332 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10472,"thread_start_us":10713,"threads_started":1}
I20260621 02:51:40.726365 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.702885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41013 (local address 127.0.0.1:50894)
0621 02:51:40.714997 (+ 12112us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.715047 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.715209 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.716220 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.716227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.716259 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:40.716942 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.716960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.719735 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.719743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.722877 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.722902 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.724707 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.724714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.724905 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.724959 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.725092 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.725295 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11864,"thread_start_us":12720,"threads_started":1}
I20260621 02:51:40.726845 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.702949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:34981 (local address 127.0.0.1:49000)
0621 02:51:40.715804 (+ 12855us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.715864 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.715988 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.716737 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.716745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.716781 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:40.717641 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.717657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.721344 (+ 3687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.721352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.723748 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.723759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.725905 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.725912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.726078 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.726107 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.726202 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.726334 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12765,"thread_start_us":11960,"threads_started":1}
I20260621 02:51:40.727476 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.702949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.11.109.2:34981)
0621 02:51:40.714800 (+ 11851us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.714811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.715951 (+ 1140us) server_negotiation.cc:407] Connection header received
0621 02:51:40.716192 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.716201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.716453 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.716906 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:40.717972 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.721121 (+ 3149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.723937 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.725450 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.725585 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.725768 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.727008 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.727199 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11645,"thread_start_us":12426,"threads_started":1}
I20260621 02:51:40.730109 22168 tablet_service.cc:1467] Processing CreateTablet for tablet 535c09ce4b5d45d998168871d10e0d31 (DEFAULT_TABLE table=test-table [id=aa2ed4ff0e054010a2f000df2a0ead81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:40.730897 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.716183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:33871 (local address 127.0.0.1:45168)
0621 02:51:40.722718 (+ 6535us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.722740 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.722855 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.724133 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.724137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.724161 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:40.724492 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.724503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.726433 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.726439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.728432 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.728445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.730183 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.730189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.730348 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.730391 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.730483 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.730597 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6408,"thread_start_us":6759,"threads_started":1}
I20260621 02:51:40.731033 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.716305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.11.109.3:33871)
0621 02:51:40.723518 (+ 7213us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.723525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.723552 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:51:40.723630 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.723637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.723854 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.724162 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:40.724666 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.726254 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.728592 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.729982 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.730110 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.730310 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.730539 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.730687 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7053,"thread_start_us":6900,"threads_started":1}
I20260621 02:51:40.730820 22302 tablet_service.cc:1467] Processing CreateTablet for tablet 535c09ce4b5d45d998168871d10e0d31 (DEFAULT_TABLE table=test-table [id=aa2ed4ff0e054010a2f000df2a0ead81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:40.731635 22168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535c09ce4b5d45d998168871d10e0d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:40.731987 22302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535c09ce4b5d45d998168871d10e0d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:40.734608 22435 tablet_service.cc:1467] Processing CreateTablet for tablet 535c09ce4b5d45d998168871d10e0d31 (DEFAULT_TABLE table=test-table [id=aa2ed4ff0e054010a2f000df2a0ead81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:51:40.735877 22435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535c09ce4b5d45d998168871d10e0d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:51:40.743645 22518 tablet_bootstrap.cc:492] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Bootstrap starting.
I20260621 02:51:40.745754 22519 tablet_bootstrap.cc:492] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Bootstrap starting.
I20260621 02:51:40.746694 22518 tablet_bootstrap.cc:654] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:40.747970 22518 log.cc:824] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:40.748760 22520 tablet_bootstrap.cc:492] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Bootstrap starting.
I20260621 02:51:40.749158 22519 tablet_bootstrap.cc:654] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:40.750696 22519 log.cc:824] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:40.751461 22520 tablet_bootstrap.cc:654] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Neither blocks nor log segments found. Creating new log.
I20260621 02:51:40.752753 22520 log.cc:824] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Log is configured to *not* fsync() on all Append() calls
I20260621 02:51:40.753818 22519 tablet_bootstrap.cc:492] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: No bootstrap required, opened a new log
I20260621 02:51:40.754069 22519 ts_tablet_manager.cc:1397] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260621 02:51:40.755051 22520 tablet_bootstrap.cc:492] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: No bootstrap required, opened a new log
I20260621 02:51:40.755275 22520 ts_tablet_manager.cc:1397] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260621 02:51:40.758843 22518 tablet_bootstrap.cc:492] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: No bootstrap required, opened a new log
I20260621 02:51:40.759083 22518 ts_tablet_manager.cc:1397] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260621 02:51:40.761036 22519 raft_consensus.cc:348] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.761469 22519 raft_consensus.cc:374] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:40.761590 22519 raft_consensus.cc:729] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff79592c8b047ef85081f738c9d8f2b, State: Initialized, Role: FOLLOWER
I20260621 02:51:40.762208 22519 consensus_queue.cc:260] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.763301 22367 heartbeater.cc:503] Master 127.11.109.62:37505 was elected leader, sending a full tablet report...
I20260621 02:51:40.763828 22519 ts_tablet_manager.cc:1428] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260621 02:51:40.765475 22520 raft_consensus.cc:348] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.765863 22520 raft_consensus.cc:374] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:40.765957 22520 raft_consensus.cc:729] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811894b09fae47e5bf6732c114587b89, State: Initialized, Role: FOLLOWER
I20260621 02:51:40.766584 22520 consensus_queue.cc:260] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.767642 22500 heartbeater.cc:503] Master 127.11.109.62:37505 was elected leader, sending a full tablet report...
I20260621 02:51:40.768347 22520 ts_tablet_manager.cc:1428] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260621 02:51:40.768916 22518 raft_consensus.cc:348] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.769387 22518 raft_consensus.cc:374] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:51:40.769528 22518 raft_consensus.cc:729] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfde5de5891c43959e2df726c5561198, State: Initialized, Role: FOLLOWER
I20260621 02:51:40.770113 22518 consensus_queue.cc:260] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.771696 22518 ts_tablet_manager.cc:1428] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260621 02:51:40.779873 22525 raft_consensus.cc:482] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:51:40.780110 22525 raft_consensus.cc:504] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.781628 22525 leader_election.cc:288] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bff79592c8b047ef85081f738c9d8f2b (127.11.109.2:34981), cfde5de5891c43959e2df726c5561198 (127.11.109.1:41013)
I20260621 02:51:40.786993 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.782093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54103 (local address 127.11.109.1:41013)
0621 02:51:40.782272 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.782278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.782295 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:51:40.782427 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.782434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.782491 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.782585 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:40.782949 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.783780 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.785400 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.786149 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.786320 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.786389 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.786643 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.786758 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:51:40.787021 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.781881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:41013 (local address 127.11.109.3:54103)
0621 02:51:40.782145 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.782168 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.782280 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.782602 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.782607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.782637 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:40.782819 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.782827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.783904 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.783912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.785250 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.785267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.786307 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.786313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.786494 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.786516 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.786620 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.786756 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260621 02:51:40.787679 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.781982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:32985 (local address 127.11.109.2:34981)
0621 02:51:40.782171 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.782179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.782853 (+ 674us) server_negotiation.cc:407] Connection header received
0621 02:51:40.783031 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.783038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.783095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.783186 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:51:40.783659 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.784324 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.786032 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.786783 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.786962 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.787029 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.787317 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.787440 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:51:40.787842 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.781996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:34981 (local address 127.11.109.3:32985)
0621 02:51:40.782746 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0621 02:51:40.782764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:51:40.782903 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:51:40.783198 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:51:40.783203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:51:40.783223 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:51:40.783518 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.783526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.784481 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.784486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.785884 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:51:40.785895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.786989 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.786995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:51:40.787170 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.787200 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:51:40.787285 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 02:51:40.787410 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":25,"thread_start_us":417,"threads_started":1}
I20260621 02:51:40.787919 22188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535c09ce4b5d45d998168871d10e0d31" candidate_uuid: "811894b09fae47e5bf6732c114587b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde5de5891c43959e2df726c5561198" is_pre_election: true
I20260621 02:51:40.788456 22188 raft_consensus.cc:2454] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811894b09fae47e5bf6732c114587b89 in term 0.
I20260621 02:51:40.788746 22322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535c09ce4b5d45d998168871d10e0d31" candidate_uuid: "811894b09fae47e5bf6732c114587b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff79592c8b047ef85081f738c9d8f2b" is_pre_election: true
I20260621 02:51:40.789090 22390 leader_election.cc:302] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 811894b09fae47e5bf6732c114587b89, cfde5de5891c43959e2df726c5561198; no voters:
I20260621 02:51:40.789218 22322 raft_consensus.cc:2454] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811894b09fae47e5bf6732c114587b89 in term 0.
I20260621 02:51:40.789503 22525 raft_consensus.cc:2781] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:51:40.789659 22525 raft_consensus.cc:482] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:51:40.789781 22525 raft_consensus.cc:3037] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:40.791989 22525 raft_consensus.cc:504] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.792623 22525 leader_election.cc:288] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [CANDIDATE]: Term 1 election: Requested vote from peers bff79592c8b047ef85081f738c9d8f2b (127.11.109.2:34981), cfde5de5891c43959e2df726c5561198 (127.11.109.1:41013)
I20260621 02:51:40.792948 22322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535c09ce4b5d45d998168871d10e0d31" candidate_uuid: "811894b09fae47e5bf6732c114587b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff79592c8b047ef85081f738c9d8f2b"
I20260621 02:51:40.793155 22322 raft_consensus.cc:3037] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:40.793109 22188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535c09ce4b5d45d998168871d10e0d31" candidate_uuid: "811894b09fae47e5bf6732c114587b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfde5de5891c43959e2df726c5561198"
I20260621 02:51:40.793359 22188 raft_consensus.cc:3037] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:51:40.795416 22322 raft_consensus.cc:2454] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811894b09fae47e5bf6732c114587b89 in term 1.
I20260621 02:51:40.795538 22188 raft_consensus.cc:2454] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811894b09fae47e5bf6732c114587b89 in term 1.
I20260621 02:51:40.795852 22389 leader_election.cc:302] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 811894b09fae47e5bf6732c114587b89, bff79592c8b047ef85081f738c9d8f2b; no voters:
I20260621 02:51:40.796212 22525 raft_consensus.cc:2781] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:51:40.796680 22525 raft_consensus.cc:686] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 1 LEADER]: Becoming Leader. State: Replica: 811894b09fae47e5bf6732c114587b89, State: Running, Role: LEADER
I20260621 02:51:40.797159 22525 consensus_queue.cc:237] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } }
I20260621 02:51:40.801769 22043 catalog_manager.cc:5495] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 811894b09fae47e5bf6732c114587b89 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "811894b09fae47e5bf6732c114587b89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811894b09fae47e5bf6732c114587b89" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 33871 } health_report { overall_health: HEALTHY } } }
I20260621 02:51:40.836069 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.830928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.11.109.3:33871)
0621 02:51:40.831091 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.831095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.831112 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:51:40.831222 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.831225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.831297 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.831422 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:51:40.831844 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.832524 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.834172 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.834552 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.834641 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.835089 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:51:40.835560 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:51:40.835671 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.835811 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.835879 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 02:51:40.843906 22188 raft_consensus.cc:1261] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 1 FOLLOWER]: Refusing update from remote peer 811894b09fae47e5bf6732c114587b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:40.843931 22322 raft_consensus.cc:1261] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 1 FOLLOWER]: Refusing update from remote peer 811894b09fae47e5bf6732c114587b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:51:40.844909 22528 consensus_queue.cc:1035] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bff79592c8b047ef85081f738c9d8f2b" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 34981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:40.845244 22525 consensus_queue.cc:1035] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfde5de5891c43959e2df726c5561198" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:51:40.856761 22531 mvcc.cc:205] Tried to move back new op lower bound from 7299114192247259136 to 7299114192069226496. Current Snapshot: MvccSnapshot[applied={T|T < 7299114192247259136}]
I20260621 02:51:40.871309 22043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:40.871598 22043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:51:40.875033 22043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 405f0d2ddc14440a8e4f6fcb86028563: Sending DeleteTablet for 3 replicas of tablet 535c09ce4b5d45d998168871d10e0d31
I20260621 02:51:40.876104 22302 tablet_service.cc:1514] Processing DeleteTablet for tablet 535c09ce4b5d45d998168871d10e0d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:40 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:49000
I20260621 02:51:40.876506 22168 tablet_service.cc:1514] Processing DeleteTablet for tablet 535c09ce4b5d45d998168871d10e0d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:40 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:50894
I20260621 02:51:40.876987 22541 tablet_replica.cc:330] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: stopping tablet replica
I20260621 02:51:40.877065 22435 tablet_service.cc:1514] Processing DeleteTablet for tablet 535c09ce4b5d45d998168871d10e0d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:51:40 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:45168
I20260621 02:51:40.877591 22541 raft_consensus.cc:2229] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:40.878201 22541 raft_consensus.cc:2258] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:40.878440 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 22105
I20260621 02:51:40.881611 22543 tablet_replica.cc:330] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: stopping tablet replica
I20260621 02:51:40.882121 22543 raft_consensus.cc:2229] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:51:40.882862 22543 raft_consensus.cc:2258] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:40.885134 22543 ts_tablet_manager.cc:1905] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:40.885452 22541 ts_tablet_manager.cc:1905] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:40.894013 22541 ts_tablet_manager.cc:1918] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:40.894213 22541 log.cc:1196] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/535c09ce4b5d45d998168871d10e0d31
I20260621 02:51:40.894843 22541 ts_tablet_manager.cc:1939] T 535c09ce4b5d45d998168871d10e0d31 P bff79592c8b047ef85081f738c9d8f2b: Deleting consensus metadata
I20260621 02:51:40.895613 22542 tablet_replica.cc:330] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: stopping tablet replica
I20260621 02:51:40.896032 22542 raft_consensus.cc:2229] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:51:40.896462 22542 raft_consensus.cc:2258] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:51:40.896910 22543 ts_tablet_manager.cc:1918] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:40.897081 22543 log.cc:1196] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/535c09ce4b5d45d998168871d10e0d31
I20260621 02:51:40.897274 22029 catalog_manager.cc:4841] TS bff79592c8b047ef85081f738c9d8f2b (127.11.109.2:34981): tablet 535c09ce4b5d45d998168871d10e0d31 (table test-table [id=aa2ed4ff0e054010a2f000df2a0ead81]) successfully deleted
I20260621 02:51:40.897711 22543 ts_tablet_manager.cc:1939] T 535c09ce4b5d45d998168871d10e0d31 P 811894b09fae47e5bf6732c114587b89: Deleting consensus metadata
I20260621 02:51:40.899386 22542 ts_tablet_manager.cc:1905] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:51:40.899513 22030 catalog_manager.cc:4841] TS 811894b09fae47e5bf6732c114587b89 (127.11.109.3:33871): tablet 535c09ce4b5d45d998168871d10e0d31 (table test-table [id=aa2ed4ff0e054010a2f000df2a0ead81]) successfully deleted
Jun 21 02:51:40 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:51:40.906443 22542 ts_tablet_manager.cc:1918] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:51:40.906611 22542 log.cc:1196] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/535c09ce4b5d45d998168871d10e0d31
I20260621 02:51:40.907169 22542 ts_tablet_manager.cc:1939] T 535c09ce4b5d45d998168871d10e0d31 P cfde5de5891c43959e2df726c5561198: Deleting consensus metadata
I20260621 02:51:40.908951 22030 catalog_manager.cc:4841] TS cfde5de5891c43959e2df726c5561198 (127.11.109.1:41013): tablet 535c09ce4b5d45d998168871d10e0d31 (table test-table [id=aa2ed4ff0e054010a2f000df2a0ead81]) successfully deleted
I20260621 02:51:40.912590 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:40.879651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.11.109.1:41013)
0621 02:51:40.879831 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:40.879839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:40.897322 (+ 17483us) server_negotiation.cc:407] Connection header received
0621 02:51:40.897620 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:40.897627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:40.897727 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:40.897812 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:40.900047 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.900837 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.902305 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:40.902702 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:40.902864 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:40.906781 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:40.906791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:40.906805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:40.906836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:40.909951 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.910724 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.910730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.910732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.910863 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:40.911272 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:40.911275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:40.911277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:40.911603 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:40.911816 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:40.912133 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 02:51:40.912277 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:51:40.913314 22228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:41.050104 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 22105
I20260621 02:51:41.072566 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 22239
Jun 21 02:51:41 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:51:41.090022 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:41.073385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.11.109.2:34981)
0621 02:51:41.073589 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:41.073594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:41.073637 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:51:41.073842 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:41.073845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:41.073919 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:41.074000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:41.075543 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:41.076397 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:41.078398 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:41.078780 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:41.078879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:41.083619 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:41.083635 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:41.083657 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:41.083695 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:41.087157 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:41.087827 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:41.087837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:41.087841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:41.087985 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:41.088332 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:41.088338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:41.088342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:41.088815 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:41.089100 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:41.089464 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 02:51:41.089668 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 02:51:41.090725 22362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:41.177487 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 22239
I20260621 02:51:41.199786 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 22372
Jun 21 02:51:41 dist-test-slave-bc52 krb5kdc[21999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:51:41.217525 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:51:41.200608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.11.109.3:33871)
0621 02:51:41.200760 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0621 02:51:41.200764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:51:41.200846 (+ 82us) server_negotiation.cc:407] Connection header received
0621 02:51:41.201019 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:51:41.201022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:51:41.201138 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:51:41.201283 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:51:41.202955 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:41.203709 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:41.205530 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:51:41.205953 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:51:41.206040 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:51:41.211445 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:51:41.211455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:51:41.211469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:51:41.211508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:51:41.214432 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:41.215148 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:41.215154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:41.215156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:41.215302 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:51:41.215723 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:51:41.215726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:51:41.215728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:51:41.216096 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:51:41.216321 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 02:51:41.216696 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 02:51:41.216961 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:51:41.218250 22495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:41.335258 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 22372
I20260621 02:51:41.353870 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 22012
I20260621 02:51:41.354918 22074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:51:41.460238 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 22012
2026-06-21T02:51:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4138 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:51:41 dist-test-slave-bc52 krb5kdc[22551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:51:41 dist-test-slave-bc52 krb5kdc[22551](info): set up 2 sockets
Jun 21 02:51:41 dist-test-slave-bc52 krb5kdc[22551](info): commencing operation
krb5kdc: starting...
W20260621 02:51:42.542794 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:51:42 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:51:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/rangeradmin_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/rangeradmin_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/rangerlookup_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/rangerlookup_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
I20260621 02:51:42.836715 11700 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSYTxIs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/postgres -l logfile start
2026-06-21 02:51:45.260 UTC [22581] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 02:51:45.260 UTC [22581] LOG: listening on IPv4 address "127.11.109.20", port 46577
2026-06-21 02:51:45.263 UTC [22581] LOG: listening on Unix socket "/tmp/.s.PGSQL.46577"
2026-06-21 02:51:45.285 UTC [22584] LOG: database system was shut down at 2026-06-21 02:51:43 UTC
2026-06-21 02:51:45.289 UTC [22581] LOG: database system is ready to accept connections
I20260621 02:51:46.265708 11700 mini_postgres.cc:96] Postgres bound to 46577
2026-06-21 02:51:46.282 UTC [22592] FATAL: database "slave" does not exist
127.11.109.20:46577 - accepting connections
I20260621 02:51:46.283090 11700 mini_ranger.cc:161] Starting Ranger...
I20260621 02:51:46.306957 11700 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 02:51:46.788635 11700 mini_ranger.cc:87] Created ranger Postgres database
I20260621 02:51:46.788772 11700 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-admin
2026-06-21 02:51:47,088 [I] DB FLAVOR :POSTGRES
2026-06-21 02:51:47,088 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:51:47,088 [I] Checking connection..
2026-06-21 02:51:47,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 02:51:47,321 [I] Checking connection passed.
2026-06-21 02:51:47,321 [I] --------- Verifying version history table ---------
2026-06-21 02:51:47,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:51:47,557 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:51:47,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:51:47,785 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:51:47,786 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 02:51:47,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 02:51:48.038 UTC [22656] WARNING: there is no transaction in progress
2026-06-21 02:51:48,045 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 02:51:48,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:51:48,293 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 02:51:48,294 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:51:48,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 02:51:48,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 02:51:48,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bc52.c.gcp-upstream.internal','N') ;"
2026-06-21 02:51:48,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:51:49,226 [I] Table x_portal_user does not exist in database ranger
2026-06-21 02:51:49,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:51:49,460 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 02:51:49,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:51:49,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:51:49,948 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 02:51:49,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 02:51:50.852 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.868 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.883 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.898 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.913 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.928 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.978 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.984 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:50.997 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.007 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.016 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.029 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.039 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.045 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.051 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51.058 UTC [22764] WARNING: there is no transaction in progress
2026-06-21 02:51:51,668 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 02:51:51,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 02:51:51,909 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 02:51:51,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:51:52,140 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 02:51:52,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:51:52,394 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 02:51:52,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:51:52,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:51:52,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bc52.c.gcp-upstream.internal';"
2026-06-21 02:51:53,109 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 02:51:53,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 02:51:53,348 [I] DB_PATCHES have already been applied
I20260621 02:51:53.354398 11700 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSYTxIs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 02:51:53.354478 11700 mini_ranger.cc:193] Using host: 127.11.109.20
I20260621 02:51:53.368431 11700 mini_ranger.cc:227] Ranger admin URL: http://127.11.109.20:51739
Jun 21, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 2:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 2:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 02:51:54.486825 11700 mini_ranger.cc:160] Time spent starting Ranger: real 8.204s user 0.005s sys 0.030s
Jun 21, 2026 2:52:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 2:52:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 21, 2026 2:52:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 2:52:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 2:52:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 02:52:14.627168 11700 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/rangerkms_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/rangerkms_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 02:52:14 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010334, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 02:52:14.774776 11700 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 02:52:14.794886 11700 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 02:52:15.643079 11700 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 02:52:15.643249 11700 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms
I20260621 02:52:15.643397 11700 mini_ranger_kms.cc:227] Using postgres at 127.11.109.20:46577
2026-06-21 02:52:16,053 [I] DB FLAVOR :POSTGRES
2026-06-21 02:52:16,053 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:52:16,054 [I] Checking connection
2026-06-21 02:52:16,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 02:52:16,275 [I] connection success
2026-06-21 02:52:16,275 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 02:52:16,275 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 02:52:16,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 02:52:16,511 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 02:52:16,511 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:52:16,511 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 02:52:16,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:46577/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 02:52:16,783 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 02:52:17.236248 11700 mini_ranger_kms.cc:310] Created kms service
I20260621 02:52:17.470191 11700 mini_ranger_kms.cc:326] Created kudu user
I20260621 02:52:17.514639 11700 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 02:52:17 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010334, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:52:17.910108 11700 mini_ranger_kms.cc:384] Added ranger policy
I20260621 02:52:17.910436 11700 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 02:52:17.910501 11700 mini_ranger_kms.cc:240] Using host: 127.11.109.20
I20260621 02:52:17.931818 11700 mini_ranger_kms.cc:276] Ranger KMS PID: 22944
I20260621 02:52:17.932006 11700 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.109.20:40613
Jun 21, 2026 2:52:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:52:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:52:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 2:52:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 2:52:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 2:52:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 02:52:20 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010340, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:52:20.229724 11700 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.455s user 0.117s sys 0.304s
I20260621 02:52:20.229980 11700 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 02:52:20.230188 11700 mini_ranger_kms.cc:401] 127.11.109.20:40613/kms/v1/keys
Jun 21 02:52:20 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010340, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.109.20@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
Jun 21 02:52:21 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010340, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.109.20@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
Jun 21 02:52:23 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:52:23.184476 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:34149
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:34149
--ranger_config_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:40613/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:52:23.506669 22995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:52:23.506983 22995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:52:23.507138 22995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:52:23.514039 22995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:52:23.514166 22995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:52:23.514261 22995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:52:23.514365 22995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:52:23.514449 22995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:52:23.524272 22995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.109.20:40613/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:34149
--ranger_config_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:52:23.525535 22995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:52:23.526777 22995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:52:23.538496 23001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:23.538466 23003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:52:23.538667 22995 server_base.cc:1034] running on GCE node
W20260621 02:52:23.538834 23000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:52:23.539664 22995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:52:23.540962 22995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:52:23.542174 22995 hybrid_clock.cc:648] HybridClock initialized: now 1782010343542130 us; error 36 us; skew 500 ppm
I20260621 02:52:23.542528 22995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:52:23 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:52:23.549017 22995 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:52:23.550498 22995 webserver.cc:460] Webserver started at http://127.11.109.62:35853/ using document root <none> and password file <none>
I20260621 02:52:23.551069 22995 fs_manager.cc:359] Metadata directory not provided
I20260621 02:52:23.551160 22995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:52:23.551395 22995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:52:23 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:52:23.598297 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "95cefa25038944bd9c0d38bb3a0349c5"
format_stamp: "Formatted at 2026-06-21 02:52:23 on dist-test-slave-bc52"
server_key: "44208a6b72e9dd5abbf98e6038ca351f"
server_key_iv: "eee553d2b8d1c2a07e9a6d168eb31b63"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:23.599114 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "95cefa25038944bd9c0d38bb3a0349c5"
format_stamp: "Formatted at 2026-06-21 02:52:23 on dist-test-slave-bc52"
server_key: "44208a6b72e9dd5abbf98e6038ca351f"
server_key_iv: "eee553d2b8d1c2a07e9a6d168eb31b63"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:23.604537 22995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:52:23.620316 23010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:23.621801 22995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:52:23.621986 22995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "95cefa25038944bd9c0d38bb3a0349c5"
format_stamp: "Formatted at 2026-06-21 02:52:23 on dist-test-slave-bc52"
server_key: "44208a6b72e9dd5abbf98e6038ca351f"
server_key_iv: "eee553d2b8d1c2a07e9a6d168eb31b63"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:23.622195 22995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:52:23.649744 22995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:52:23.653287 22995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:52:23.653568 22995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:52:23.671972 22995 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:34149
I20260621 02:52:23.672010 23061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:34149 every 8 connection(s)
I20260621 02:52:23.673277 22995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:52:23.676582 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 22995
I20260621 02:52:23.676666 23062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:52:23.676801 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:52:23.677495 11700 external_mini_cluster.cc:1428] Setting key 6e0aa04158c3f77091d3a44a12e01f35
I20260621 02:52:23.689947 23062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5: Bootstrap starting.
Jun 21 02:52:23 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:52:23.692704 23062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5: Neither blocks nor log segments found. Creating new log.
I20260621 02:52:23.693876 23062 log.cc:824] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:52:23.696841 23062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5: No bootstrap required, opened a new log
I20260621 02:52:23.700625 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:23.679294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.11.109.62:34149)
0621 02:52:23.680357 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:23.680368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:23.680405 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:52:23.682620 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:23.682645 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:23.683026 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:23.683334 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:23.684924 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:23.686989 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:23.688228 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:23.688906 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:23.689079 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:23.692834 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:23.692846 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:23.692869 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:23.692916 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:23.696368 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:23.697076 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:23.697085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:23.697096 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:23.697325 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:23.697675 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:23.697684 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:23.697687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:23.698315 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:23.698614 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:23.698916 (+ 302us) server_negotiation.cc:299] Negotiation successful
0621 02:52:23.699326 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":668,"threads_started":1}
I20260621 02:52:23.705283 23062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95cefa25038944bd9c0d38bb3a0349c5" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34149 } }
I20260621 02:52:23.705572 23062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:52:23.705669 23062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95cefa25038944bd9c0d38bb3a0349c5, State: Initialized, Role: FOLLOWER
I20260621 02:52:23.706197 23062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95cefa25038944bd9c0d38bb3a0349c5" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34149 } }
I20260621 02:52:23.706455 23062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:52:23.706601 23062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:52:23.706765 23062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:52:23.709080 23062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95cefa25038944bd9c0d38bb3a0349c5" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34149 } }
I20260621 02:52:23.709523 23062 leader_election.cc:302] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95cefa25038944bd9c0d38bb3a0349c5; no voters:
I20260621 02:52:23.710062 23062 leader_election.cc:288] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:52:23.710206 23067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:52:23.711024 23067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [term 1 LEADER]: Becoming Leader. State: Replica: 95cefa25038944bd9c0d38bb3a0349c5, State: Running, Role: LEADER
I20260621 02:52:23.711838 23067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95cefa25038944bd9c0d38bb3a0349c5" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34149 } }
I20260621 02:52:23.712222 23062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:52:23.714481 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95cefa25038944bd9c0d38bb3a0349c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95cefa25038944bd9c0d38bb3a0349c5" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34149 } } }
I20260621 02:52:23.714830 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:52:23.714704 23069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95cefa25038944bd9c0d38bb3a0349c5. Latest consensus state: current_term: 1 leader_uuid: "95cefa25038944bd9c0d38bb3a0349c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95cefa25038944bd9c0d38bb3a0349c5" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 34149 } } }
I20260621 02:52:23.715051 23069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:52:23.718271 23075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:52:23.723258 23075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:52:23.724295 23062 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:52:23.740676 23075 catalog_manager.cc:1347] Generated new cluster ID: 7bdfc226b0c94a8894e4cab400cf9708
I20260621 02:52:23.740823 23075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:52:23.774701 23075 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:52:23.776646 23075 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:52:23.791045 23075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5: Generated new TSK 0
I20260621 02:52:23.792073 23075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:52:24.492561 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:40613/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34149
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:52:24 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 02:52:24.911024 23107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:52:24.911439 23107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:52:24.911600 23107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:52:24.923436 23107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:52:24.923566 23107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:52:24.923766 23107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:52:24.936133 23107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.109.20:40613/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:52:24.937856 23107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:52:24.939478 23107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:52:24.958014 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:24.959544 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:24.963519 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:52:24.966903 23107 server_base.cc:1034] running on GCE node
I20260621 02:52:24.967720 23107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:52:24.969715 23107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:52:24.971016 23107 hybrid_clock.cc:648] HybridClock initialized: now 1782010344970958 us; error 62 us; skew 500 ppm
I20260621 02:52:24.971405 23107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:52:24 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:52:24.975124 23107 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:52:24.977057 23107 webserver.cc:460] Webserver started at http://127.11.109.1:33633/ using document root <none> and password file <none>
I20260621 02:52:24.977771 23107 fs_manager.cc:359] Metadata directory not provided
I20260621 02:52:24.977900 23107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:52:24.978212 23107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:52:24 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:52:24.995430 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "3b90934c1c574e96a927a0d015371434"
format_stamp: "Formatted at 2026-06-21 02:52:24 on dist-test-slave-bc52"
server_key: "c8ffc1232bcface81a909166451c374f"
server_key_iv: "34bf80ff7c5932be03591ec3f76dbf5d"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:24.996425 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "3b90934c1c574e96a927a0d015371434"
format_stamp: "Formatted at 2026-06-21 02:52:24 on dist-test-slave-bc52"
server_key: "c8ffc1232bcface81a909166451c374f"
server_key_iv: "34bf80ff7c5932be03591ec3f76dbf5d"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:25.002902 23107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260621 02:52:25.033787 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:25.035748 23107 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.008s
I20260621 02:52:25.035945 23107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "3b90934c1c574e96a927a0d015371434"
format_stamp: "Formatted at 2026-06-21 02:52:24 on dist-test-slave-bc52"
server_key: "c8ffc1232bcface81a909166451c374f"
server_key_iv: "34bf80ff7c5932be03591ec3f76dbf5d"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:25.036190 23107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:52:25.078464 23107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:52:25.083691 23107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:52:25.084523 23107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:52:25.085795 23107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:52:25.087474 23107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:52:25.087569 23107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:25.087736 23107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:52:25.087805 23107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:52:25 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:52:25.277215 23107 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:45711
I20260621 02:52:25.277294 23239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:45711 every 8 connection(s)
I20260621 02:52:25.278918 23107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:52:25.279915 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 23107
I20260621 02:52:25.280110 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:52:25.280874 11700 external_mini_cluster.cc:1428] Setting key e2d5eb0901e586c230babb4c6f361d65
Jun 21 02:52:25 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
I20260621 02:52:25.313622 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:25.290786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:45435 (local address 127.11.109.62:34149)
0621 02:52:25.294838 (+ 4052us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:25.294842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:25.294859 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:52:25.294908 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:25.294912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:25.294978 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:25.295261 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:25.296843 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.297974 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.300725 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.301515 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.301723 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:25.306558 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:25.306567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:25.306575 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:25.306622 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:25.310425 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:25.311129 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:25.311135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:25.311139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:25.311266 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:25.311718 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:25.311725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:25.311729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:25.312027 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:25.312165 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:25.312885 (+ 720us) server_negotiation.cc:299] Negotiation successful
0621 02:52:25.313106 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3951,"thread_start_us":3855,"threads_started":1}
I20260621 02:52:25.314616 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:25.285191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34149 (local address 127.11.109.1:45435)
0621 02:52:25.290043 (+ 4852us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:25.290104 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:25.293245 (+ 3141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:25.295528 (+ 2283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:25.295537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:25.295997 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:52:25.296577 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:25.296593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.298170 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.298179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:25.300526 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:25.300558 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.302032 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.302042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:25.302390 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:25.303216 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:52:25.303253 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:52:25.306278 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:52:25.310560 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:52:25.310569 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:52:25.310585 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:52:25.311008 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:52:25.311387 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:52:25.311392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:52:25.311396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:52:25.311604 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:52:25.312159 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:52:25.312166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:52:25.312543 (+ 377us) client_negotiation.cc:769] Sending connection context
0621 02:52:25.312803 (+ 260us) client_negotiation.cc:240] Negotiation successful
0621 02:52:25.313406 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4434,"thread_start_us":4197,"threads_started":1}
I20260621 02:52:25.316219 23240 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34149
I20260621 02:52:25.316529 23240 heartbeater.cc:464] Registering TS with master...
I20260621 02:52:25.317355 23240 heartbeater.cc:511] Master 127.11.109.62:34149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:52:25.319819 23027 ts_manager.cc:194] Registered new tserver with Master: 3b90934c1c574e96a927a0d015371434 (127.11.109.1:45711)
I20260621 02:52:25.321734 23027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:45435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:52:25.354230 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:40613/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34149
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:52:25 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010344, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
W20260621 02:52:25.808386 23246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:52:25.808640 23246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:52:25.808782 23246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:52:25.815850 23246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:52:25.815956 23246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:52:25.816119 23246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:52:25.825843 23246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.109.20:40613/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:52:25.827051 23246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:52:25.828303 23246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:52:25.840747 23260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:25.840770 23263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:25.841289 23261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:52:25.842015 23246 server_base.cc:1034] running on GCE node
I20260621 02:52:25.842628 23246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:52:25.843474 23246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:52:25.844662 23246 hybrid_clock.cc:648] HybridClock initialized: now 1782010345844617 us; error 35 us; skew 500 ppm
I20260621 02:52:25.844960 23246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:52:25 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:52:25.847898 23246 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:52:25.849331 23246 webserver.cc:460] Webserver started at http://127.11.109.2:45455/ using document root <none> and password file <none>
I20260621 02:52:25.849974 23246 fs_manager.cc:359] Metadata directory not provided
I20260621 02:52:25.850083 23246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:52:25.850412 23246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:52:25 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:52:25.864084 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "e7d37724a2bc4214889d875ca0e64b33"
format_stamp: "Formatted at 2026-06-21 02:52:25 on dist-test-slave-bc52"
server_key: "d9f23288a935e14e97489291647023cc"
server_key_iv: "84b8b231b1f4374b7ab748b46d3c7127"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:25.864923 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "e7d37724a2bc4214889d875ca0e64b33"
format_stamp: "Formatted at 2026-06-21 02:52:25 on dist-test-slave-bc52"
server_key: "d9f23288a935e14e97489291647023cc"
server_key_iv: "84b8b231b1f4374b7ab748b46d3c7127"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:25.869792 23246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:52:25.885452 23270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:25.886967 23246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:52:25.887131 23246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "e7d37724a2bc4214889d875ca0e64b33"
format_stamp: "Formatted at 2026-06-21 02:52:25 on dist-test-slave-bc52"
server_key: "d9f23288a935e14e97489291647023cc"
server_key_iv: "84b8b231b1f4374b7ab748b46d3c7127"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:25.887328 23246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:52:25.900718 23246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:52:25.904407 23246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:52:25.904704 23246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:52:25.905640 23246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:52:25.907333 23246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:52:25.907409 23246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:25.907531 23246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:52:25.907588 23246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:25.959229 23246 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:36063
I20260621 02:52:25.959299 23382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:36063 every 8 connection(s)
I20260621 02:52:25.960472 23246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:52:25.965718 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 23246
I20260621 02:52:25.965888 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:52:25.966456 11700 external_mini_cluster.cc:1428] Setting key f3d818a2831fcb64bd62b8bb4e5a09e6
Jun 21 02:52:25 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010345, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:52:25.985606 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:25.963323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:43009 (local address 127.11.109.62:34149)
0621 02:52:25.963867 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:25.963874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:25.964827 (+ 953us) server_negotiation.cc:407] Connection header received
0621 02:52:25.966913 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:25.966919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:25.967045 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:25.967162 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:25.969050 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.970345 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.973091 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.973880 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.974018 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:25.978042 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:25.978051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:25.978059 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:25.978103 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:25.981870 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:25.982759 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:25.982768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:25.982775 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:25.982921 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:25.983403 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:25.983410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:25.983416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:25.983796 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:25.983933 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:25.984797 (+ 864us) server_negotiation.cc:299] Negotiation successful
0621 02:52:25.985054 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":273,"threads_started":1}
I20260621 02:52:25.986811 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:25.963890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34149 (local address 127.11.109.2:43009)
0621 02:52:25.964651 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:25.964695 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:25.966640 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:25.967441 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:25.967453 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:25.968072 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:52:25.968850 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:25.968876 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.970929 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.970935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:25.972927 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:25.972952 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:25.974115 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:25.974123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:25.974415 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:25.975204 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:52:25.975234 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:52:25.977782 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:52:25.982014 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:52:25.982023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:52:25.982041 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:52:25.982632 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:52:25.983051 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:52:25.983056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:52:25.983060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:52:25.983278 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:52:25.983963 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:52:25.983972 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:52:25.984368 (+ 396us) client_negotiation.cc:769] Sending connection context
0621 02:52:25.984688 (+ 320us) client_negotiation.cc:240] Negotiation successful
0621 02:52:25.985263 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":321,"threads_started":1}
I20260621 02:52:25.988493 23383 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34149
I20260621 02:52:25.988821 23383 heartbeater.cc:464] Registering TS with master...
I20260621 02:52:25.989735 23383 heartbeater.cc:511] Master 127.11.109.62:34149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:52:25.991479 23027 ts_manager.cc:194] Registered new tserver with Master: e7d37724a2bc4214889d875ca0e64b33 (127.11.109.2:36063)
I20260621 02:52:25.992637 23027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:43009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:52:26.022557 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:40613/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34149
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:52:26.325307 23240 heartbeater.cc:503] Master 127.11.109.62:34149 was elected leader, sending a full tablet report...
W20260621 02:52:26.352675 23389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:52:26.353012 23389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:52:26.353211 23389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:52:26.360551 23389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:52:26.360642 23389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:52:26.360889 23389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:52:26.370682 23389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.109.20:40613/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:34149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:52:26.371884 23389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:52:26.373099 23389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:52:26.386950 23397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:26.387127 23394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:52:26.387053 23395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:52:26.388116 23389 server_base.cc:1034] running on GCE node
I20260621 02:52:26.388682 23389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:52:26.389634 23389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:52:26.390849 23389 hybrid_clock.cc:648] HybridClock initialized: now 1782010346390785 us; error 75 us; skew 500 ppm
I20260621 02:52:26.391181 23389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:52:26 dist-test-slave-bc52 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010346, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:52:26.394371 23389 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:52:26.395699 23389 webserver.cc:460] Webserver started at http://127.11.109.3:40345/ using document root <none> and password file <none>
I20260621 02:52:26.396346 23389 fs_manager.cc:359] Metadata directory not provided
I20260621 02:52:26.396495 23389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:52:26.396785 23389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:52:26 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010346, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:52:26.413409 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "00f8b087ef244e83bf06a12e0db9dd77"
format_stamp: "Formatted at 2026-06-21 02:52:26 on dist-test-slave-bc52"
server_key: "e1a3ad29f6dcf3366b62043841dc378e"
server_key_iv: "8f484c25a8f6e4a25466f452247bb8b9"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:26.414499 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "00f8b087ef244e83bf06a12e0db9dd77"
format_stamp: "Formatted at 2026-06-21 02:52:26 on dist-test-slave-bc52"
server_key: "e1a3ad29f6dcf3366b62043841dc378e"
server_key_iv: "8f484c25a8f6e4a25466f452247bb8b9"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:26.421139 23389 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.006s
I20260621 02:52:26.438006 23404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:26.439795 23389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:52:26.439968 23389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "00f8b087ef244e83bf06a12e0db9dd77"
format_stamp: "Formatted at 2026-06-21 02:52:26 on dist-test-slave-bc52"
server_key: "e1a3ad29f6dcf3366b62043841dc378e"
server_key_iv: "8f484c25a8f6e4a25466f452247bb8b9"
server_key_version: "kuduclusterkey@0"
I20260621 02:52:26.440201 23389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:52:26.469839 23389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:52:26.473444 23389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:52:26.473680 23389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:52:26.474517 23389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:52:26.475783 23389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:52:26.475857 23389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:26.475934 23389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:52:26.475976 23389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:52:26.509464 23389 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:36189
I20260621 02:52:26.509544 23516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:36189 every 8 connection(s)
I20260621 02:52:26.510695 23389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:52:26.515972 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 23389
I20260621 02:52:26.516134 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:52:26.516569 11700 external_mini_cluster.cc:1428] Setting key cb898703dcf6d91c41482e126bf61da4
Jun 21 02:52:26 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010346, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:52:26.530330 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.513138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:39565 (local address 127.11.109.62:34149)
0621 02:52:26.513597 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.513602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.514950 (+ 1348us) server_negotiation.cc:407] Connection header received
0621 02:52:26.516835 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.516840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.516931 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.517040 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:26.518713 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.519531 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.521014 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.521568 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.521698 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.524763 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:26.524768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:26.524772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:26.524800 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:26.527506 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.528204 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.528207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.528210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.528325 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.528670 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.528673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.528675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.528915 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:26.529027 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.529719 (+ 692us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.529920 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":225,"threads_started":1}
I20260621 02:52:26.531438 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.513763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:34149 (local address 127.11.109.3:39565)
0621 02:52:26.514796 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:26.514839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:26.516519 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:26.517337 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:26.517348 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:26.517905 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:52:26.518559 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.518579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.519668 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.519673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.520875 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.520887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.521778 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.521786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.522017 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.522608 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:52:26.522630 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:52:26.524529 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:52:26.527654 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:52:26.527661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:52:26.527674 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:52:26.528100 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:52:26.528422 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:52:26.528425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:52:26.528427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:52:26.528586 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:52:26.529010 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:52:26.529015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:52:26.529372 (+ 357us) client_negotiation.cc:769] Sending connection context
0621 02:52:26.529628 (+ 256us) client_negotiation.cc:240] Negotiation successful
0621 02:52:26.530134 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":635,"threads_started":1}
I20260621 02:52:26.532735 23517 heartbeater.cc:347] Connected to a master server at 127.11.109.62:34149
I20260621 02:52:26.533035 23517 heartbeater.cc:464] Registering TS with master...
I20260621 02:52:26.534143 23517 heartbeater.cc:511] Master 127.11.109.62:34149 requested a full tablet report, sending...
I20260621 02:52:26.535683 23027 ts_manager.cc:194] Registered new tserver with Master: 00f8b087ef244e83bf06a12e0db9dd77 (127.11.109.3:36189)
I20260621 02:52:26.536576 23027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:39565
I20260621 02:52:26.543828 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:52:26.560137 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.548353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.11.109.62:34149)
0621 02:52:26.548509 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.548514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.548942 (+ 428us) server_negotiation.cc:407] Connection header received
0621 02:52:26.549077 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.549081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.549152 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.549274 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:26.550525 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.551214 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.552467 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.552863 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.552964 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.555425 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:26.555430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:26.555433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:26.555462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:26.557974 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.558576 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.558580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.558582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.558695 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.558999 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.559002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.559004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.559232 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:26.559403 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.559640 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.559796 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:52:26.563884 23027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:52:26.571206 23027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:52:26.596123 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.582268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45711 (local address 127.0.0.1:55174)
0621 02:52:26.583278 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:26.583324 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:26.583542 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:26.585471 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:26.585478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:26.585513 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:52:26.586069 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.586086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.588797 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.588806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.592095 (+ 3289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.592114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.594593 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.594603 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.594878 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.594968 (+ 90us) client_negotiation.cc:769] Sending connection context
0621 02:52:26.595113 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 02:52:26.595359 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":56,"thread_start_us":863,"threads_started":1}
I20260621 02:52:26.596627 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.582382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36063 (local address 127.0.0.1:59124)
0621 02:52:26.585512 (+ 3130us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:26.585535 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:26.585662 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:26.586825 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:26.586833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:26.586868 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:52:26.587274 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.587288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.589920 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.589929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.592750 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.592770 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.595107 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.595113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.595314 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.595353 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:52:26.595431 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 02:52:26.595597 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"mutex_wait_us":3,"thread_start_us":2800,"threads_started":1}
I20260621 02:52:26.597239 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.582620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.11.109.2:36063)
0621 02:52:26.586160 (+ 3540us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.586168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.586195 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:52:26.586277 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.586283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.586575 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.587482 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:52:26.587698 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.589730 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.593008 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.594868 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.596180 (+ 1312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.596476 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.596563 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.596789 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3375,"thread_start_us":3512,"threads_started":1}
I20260621 02:52:26.598165 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.582747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.11.109.1:45711)
0621 02:52:26.584699 (+ 1952us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.584711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.584746 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:52:26.584869 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.584880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.585231 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.586324 (+ 1093us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:52:26.586601 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.588606 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.592345 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.594317 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.597252 (+ 2935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.597558 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.597638 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.597822 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":2044,"threads_started":1}
I20260621 02:52:26.602022 23175 tablet_service.cc:1467] Processing CreateTablet for tablet 45db4b42d2b74045aa84cbc83bc16d84 (DEFAULT_TABLE table=test-table [id=15d0ad6421b240f399e6cc7a625a7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:52:26.602028 23318 tablet_service.cc:1467] Processing CreateTablet for tablet 45db4b42d2b74045aa84cbc83bc16d84 (DEFAULT_TABLE table=test-table [id=15d0ad6421b240f399e6cc7a625a7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:52:26.603509 23175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45db4b42d2b74045aa84cbc83bc16d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:52:26.603509 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45db4b42d2b74045aa84cbc83bc16d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:52:26.604307 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.585591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:36189 (local address 127.0.0.1:43968)
0621 02:52:26.596840 (+ 11249us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:26.596856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:26.596967 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:26.599093 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:26.599097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:26.599122 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:52:26.599289 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.599299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.601041 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.601049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.602288 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.602299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.603556 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.603562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.603728 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.603760 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:52:26.603887 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:52:26.604013 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11159,"thread_start_us":12280,"threads_started":1}
I20260621 02:52:26.604398 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.585931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.11.109.3:36189)
0621 02:52:26.598635 (+ 12704us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.598641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.598667 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:52:26.598745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.598750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.598915 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.599113 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:52:26.599639 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.600920 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.602436 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.603395 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.603546 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.603753 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.603931 (+ 178us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.604074 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12570,"thread_start_us":11988,"threads_started":1}
I20260621 02:52:26.608158 23452 tablet_service.cc:1467] Processing CreateTablet for tablet 45db4b42d2b74045aa84cbc83bc16d84 (DEFAULT_TABLE table=test-table [id=15d0ad6421b240f399e6cc7a625a7630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:52:26.609472 23452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45db4b42d2b74045aa84cbc83bc16d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:52:26.619128 23537 tablet_bootstrap.cc:492] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Bootstrap starting.
I20260621 02:52:26.619940 23536 tablet_bootstrap.cc:492] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Bootstrap starting.
I20260621 02:52:26.622459 23538 tablet_bootstrap.cc:492] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Bootstrap starting.
I20260621 02:52:26.623834 23536 tablet_bootstrap.cc:654] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Neither blocks nor log segments found. Creating new log.
I20260621 02:52:26.624104 23537 tablet_bootstrap.cc:654] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Neither blocks nor log segments found. Creating new log.
I20260621 02:52:26.624913 23536 log.cc:824] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Log is configured to *not* fsync() on all Append() calls
I20260621 02:52:26.625775 23537 log.cc:824] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Log is configured to *not* fsync() on all Append() calls
I20260621 02:52:26.626362 23538 tablet_bootstrap.cc:654] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Neither blocks nor log segments found. Creating new log.
I20260621 02:52:26.627465 23538 log.cc:824] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Log is configured to *not* fsync() on all Append() calls
I20260621 02:52:26.628118 23536 tablet_bootstrap.cc:492] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: No bootstrap required, opened a new log
I20260621 02:52:26.628422 23536 ts_tablet_manager.cc:1397] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:52:26.628794 23537 tablet_bootstrap.cc:492] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: No bootstrap required, opened a new log
I20260621 02:52:26.629060 23537 ts_tablet_manager.cc:1397] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260621 02:52:26.630290 23538 tablet_bootstrap.cc:492] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: No bootstrap required, opened a new log
I20260621 02:52:26.630539 23538 ts_tablet_manager.cc:1397] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 02:52:26.635833 23536 raft_consensus.cc:348] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.636180 23536 raft_consensus.cc:374] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:52:26.636276 23536 raft_consensus.cc:729] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b90934c1c574e96a927a0d015371434, State: Initialized, Role: FOLLOWER
I20260621 02:52:26.636916 23536 consensus_queue.cc:260] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.639708 23536 ts_tablet_manager.cc:1428] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260621 02:52:26.640232 23537 raft_consensus.cc:348] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.640637 23537 raft_consensus.cc:374] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:52:26.640734 23537 raft_consensus.cc:729] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d37724a2bc4214889d875ca0e64b33, State: Initialized, Role: FOLLOWER
I20260621 02:52:26.641320 23537 consensus_queue.cc:260] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.641376 23538 raft_consensus.cc:348] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.641767 23538 raft_consensus.cc:374] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:52:26.641954 23538 raft_consensus.cc:729] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f8b087ef244e83bf06a12e0db9dd77, State: Initialized, Role: FOLLOWER
I20260621 02:52:26.642517 23383 heartbeater.cc:503] Master 127.11.109.62:34149 was elected leader, sending a full tablet report...
I20260621 02:52:26.643043 23537 ts_tablet_manager.cc:1428] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260621 02:52:26.643500 23538 consensus_queue.cc:260] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.645465 23517 heartbeater.cc:503] Master 127.11.109.62:34149 was elected leader, sending a full tablet report...
I20260621 02:52:26.645864 23538 ts_tablet_manager.cc:1428] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260621 02:52:26.712529 23384 tablet.cc:2369] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:52:26.741117 23544 raft_consensus.cc:482] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:52:26.741430 23544 raft_consensus.cc:504] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.743229 23544 leader_election.cc:288] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b90934c1c574e96a927a0d015371434 (127.11.109.1:45711), e7d37724a2bc4214889d875ca0e64b33 (127.11.109.2:36063)
I20260621 02:52:26.749744 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.743504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54095 (local address 127.11.109.1:45711)
0621 02:52:26.743723 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.743729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.744014 (+ 285us) server_negotiation.cc:407] Connection header received
0621 02:52:26.744247 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.744254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.744344 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.744456 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:52:26.744879 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.746056 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.747848 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.748842 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.748967 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.749019 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.749370 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.749472 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:52:26.749784 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.743648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45711 (local address 127.11.109.3:54095)
0621 02:52:26.743880 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:26.743922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:26.744074 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:26.744490 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:26.744494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:26.744518 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:52:26.744709 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.744733 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.746180 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.746185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.747691 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.747702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.748988 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.748994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.749212 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.749249 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:52:26.749350 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:52:26.749490 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260621 02:52:26.750454 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.743865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:50211 (local address 127.11.109.2:36063)
0621 02:52:26.744077 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.744085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.744614 (+ 529us) server_negotiation.cc:407] Connection header received
0621 02:52:26.744795 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.744802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.744939 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.745021 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:52:26.745642 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.746594 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.748453 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.749463 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.749607 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.749677 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.750079 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.750205 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:52:26.750473 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.743714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36063 (local address 127.11.109.3:50211)
0621 02:52:26.744504 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0621 02:52:26.744523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:52:26.744665 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:52:26.745067 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:52:26.745072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:52:26.745096 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:52:26.745482 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.745494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.746721 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.746725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.748289 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:52:26.748302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.749667 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.749672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:52:26.749865 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.749902 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:52:26.750048 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 02:52:26.750168 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":472,"threads_started":1}
I20260621 02:52:26.751241 23195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45db4b42d2b74045aa84cbc83bc16d84" candidate_uuid: "00f8b087ef244e83bf06a12e0db9dd77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90934c1c574e96a927a0d015371434" is_pre_election: true
I20260621 02:52:26.751394 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45db4b42d2b74045aa84cbc83bc16d84" candidate_uuid: "00f8b087ef244e83bf06a12e0db9dd77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7d37724a2bc4214889d875ca0e64b33" is_pre_election: true
I20260621 02:52:26.751780 23195 raft_consensus.cc:2454] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f8b087ef244e83bf06a12e0db9dd77 in term 0.
I20260621 02:52:26.751910 23338 raft_consensus.cc:2454] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f8b087ef244e83bf06a12e0db9dd77 in term 0.
I20260621 02:52:26.752514 23408 leader_election.cc:302] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f8b087ef244e83bf06a12e0db9dd77, 3b90934c1c574e96a927a0d015371434; no voters:
I20260621 02:52:26.752871 23544 raft_consensus.cc:2781] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:52:26.753031 23544 raft_consensus.cc:482] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:52:26.753130 23544 raft_consensus.cc:3037] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:52:26.755319 23544 raft_consensus.cc:504] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
I20260621 02:52:26.755851 23544 leader_election.cc:288] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [CANDIDATE]: Term 1 election: Requested vote from peers 3b90934c1c574e96a927a0d015371434 (127.11.109.1:45711), e7d37724a2bc4214889d875ca0e64b33 (127.11.109.2:36063)
I20260621 02:52:26.756275 23195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45db4b42d2b74045aa84cbc83bc16d84" candidate_uuid: "00f8b087ef244e83bf06a12e0db9dd77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90934c1c574e96a927a0d015371434"
I20260621 02:52:26.756316 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45db4b42d2b74045aa84cbc83bc16d84" candidate_uuid: "00f8b087ef244e83bf06a12e0db9dd77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7d37724a2bc4214889d875ca0e64b33"
I20260621 02:52:26.756426 23195 raft_consensus.cc:3037] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:52:26.756527 23338 raft_consensus.cc:3037] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:52:26.758599 23195 raft_consensus.cc:2454] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f8b087ef244e83bf06a12e0db9dd77 in term 1.
I20260621 02:52:26.758781 23338 raft_consensus.cc:2454] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f8b087ef244e83bf06a12e0db9dd77 in term 1.
I20260621 02:52:26.758996 23408 leader_election.cc:302] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f8b087ef244e83bf06a12e0db9dd77, 3b90934c1c574e96a927a0d015371434; no voters:
I20260621 02:52:26.759269 23544 raft_consensus.cc:2781] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:52:26.759702 23544 raft_consensus.cc:686] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 1 LEADER]: Becoming Leader. State: Replica: 00f8b087ef244e83bf06a12e0db9dd77, State: Running, Role: LEADER
I20260621 02:52:26.760152 23544 consensus_queue.cc:237] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } }
W20260621 02:52:26.762146 23518 tablet.cc:2369] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:52:26.764827 23027 catalog_manager.cc:5495] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f8b087ef244e83bf06a12e0db9dd77 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "00f8b087ef244e83bf06a12e0db9dd77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00f8b087ef244e83bf06a12e0db9dd77" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 36189 } health_report { overall_health: HEALTHY } } }
I20260621 02:52:26.784996 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.779536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.11.109.3:36189)
0621 02:52:26.779718 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.779724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.779796 (+ 72us) server_negotiation.cc:407] Connection header received
0621 02:52:26.780032 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.780036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.780117 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.780259 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:52:26.780665 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.781481 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.782988 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.783374 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.783504 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.783983 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:52:26.784458 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:52:26.784611 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.784665 (+ 54us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.784739 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260621 02:52:26.790107 23241 tablet.cc:2369] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:52:26.792870 23195 raft_consensus.cc:1261] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 1 FOLLOWER]: Refusing update from remote peer 00f8b087ef244e83bf06a12e0db9dd77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:52:26.792870 23338 raft_consensus.cc:1261] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 1 FOLLOWER]: Refusing update from remote peer 00f8b087ef244e83bf06a12e0db9dd77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:52:26.793871 23544 consensus_queue.cc:1035] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b90934c1c574e96a927a0d015371434" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:52:26.794167 23546 consensus_queue.cc:1035] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d37724a2bc4214889d875ca0e64b33" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:52:26.804958 23549 mvcc.cc:205] Tried to move back new op lower bound from 7299114380453691392 to 7299114380333363200. Current Snapshot: MvccSnapshot[applied={T|T < 7299114380453691392}]
I20260621 02:52:26.819267 23027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:52:26.819494 23027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:52:26.822649 23027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95cefa25038944bd9c0d38bb3a0349c5: Sending DeleteTablet for 3 replicas of tablet 45db4b42d2b74045aa84cbc83bc16d84
I20260621 02:52:26.823601 23175 tablet_service.cc:1514] Processing DeleteTablet for tablet 45db4b42d2b74045aa84cbc83bc16d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:52:26 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:55174
I20260621 02:52:26.823787 23318 tablet_service.cc:1514] Processing DeleteTablet for tablet 45db4b42d2b74045aa84cbc83bc16d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:52:26 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:59124
I20260621 02:52:26.823786 23452 tablet_service.cc:1514] Processing DeleteTablet for tablet 45db4b42d2b74045aa84cbc83bc16d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:52:26 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:43968
I20260621 02:52:26.825127 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 23107
I20260621 02:52:26.837736 23560 tablet_replica.cc:330] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: stopping tablet replica
I20260621 02:52:26.838240 23560 raft_consensus.cc:2229] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:52:26.838951 23560 raft_consensus.cc:2258] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:52:26.841784 23560 ts_tablet_manager.cc:1905] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:52:26.841991 23559 tablet_replica.cc:330] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: stopping tablet replica
I20260621 02:52:26.842430 23559 raft_consensus.cc:2229] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:52:26.842934 23559 raft_consensus.cc:2258] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:52:26.843677 23561 tablet_replica.cc:330] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: stopping tablet replica
I20260621 02:52:26.844066 23561 raft_consensus.cc:2229] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:52:26.844504 23561 raft_consensus.cc:2258] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:52:26.846276 23561 ts_tablet_manager.cc:1905] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:52:26.849720 23560 ts_tablet_manager.cc:1918] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 21 02:52:26 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:52:26.851459 23559 ts_tablet_manager.cc:1905] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:52:26.852905 23561 ts_tablet_manager.cc:1918] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:52:26.853066 23561 log.cc:1196] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/45db4b42d2b74045aa84cbc83bc16d84
I20260621 02:52:26.853590 23561 ts_tablet_manager.cc:1939] T 45db4b42d2b74045aa84cbc83bc16d84 P e7d37724a2bc4214889d875ca0e64b33: Deleting consensus metadata
I20260621 02:52:26.855304 23012 catalog_manager.cc:4841] TS e7d37724a2bc4214889d875ca0e64b33 (127.11.109.2:36063): tablet 45db4b42d2b74045aa84cbc83bc16d84 (table test-table [id=15d0ad6421b240f399e6cc7a625a7630]) successfully deleted
I20260621 02:52:26.857739 23560 log.cc:1196] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/45db4b42d2b74045aa84cbc83bc16d84
I20260621 02:52:26.858105 23559 ts_tablet_manager.cc:1918] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:52:26.858327 23559 log.cc:1196] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/45db4b42d2b74045aa84cbc83bc16d84
I20260621 02:52:26.858376 23560 ts_tablet_manager.cc:1939] T 45db4b42d2b74045aa84cbc83bc16d84 P 00f8b087ef244e83bf06a12e0db9dd77: Deleting consensus metadata
I20260621 02:52:26.858924 23559 ts_tablet_manager.cc:1939] T 45db4b42d2b74045aa84cbc83bc16d84 P 3b90934c1c574e96a927a0d015371434: Deleting consensus metadata
I20260621 02:52:26.860108 23012 catalog_manager.cc:4841] TS 00f8b087ef244e83bf06a12e0db9dd77 (127.11.109.3:36189): tablet 45db4b42d2b74045aa84cbc83bc16d84 (table test-table [id=15d0ad6421b240f399e6cc7a625a7630]) successfully deleted
I20260621 02:52:26.860437 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.826627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.11.109.1:45711)
0621 02:52:26.826805 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.826809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.839761 (+ 12952us) server_negotiation.cc:407] Connection header received
0621 02:52:26.839987 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.839993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.840071 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.840163 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:26.842105 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.843282 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.845670 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.846263 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.846386 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.852064 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:26.852078 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:26.852095 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:26.852132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:26.856721 (+ 4589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.857674 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.857682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.857685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.857862 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.858431 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.858436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.858439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.858902 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:26.859207 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.859786 (+ 579us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.860040 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:52:26.860589 23014 catalog_manager.cc:4841] TS 3b90934c1c574e96a927a0d015371434 (127.11.109.1:45711): tablet 45db4b42d2b74045aa84cbc83bc16d84 (table test-table [id=15d0ad6421b240f399e6cc7a625a7630]) successfully deleted
I20260621 02:52:26.861692 23235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:52:26.950778 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 23107
I20260621 02:52:26.971891 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 23246
Jun 21 02:52:26 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:52:26.991227 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:26.972711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.11.109.2:36063)
0621 02:52:26.972898 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:26.972905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:26.973168 (+ 263us) server_negotiation.cc:407] Connection header received
0621 02:52:26.973355 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:26.973360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:26.973457 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:26.973557 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:26.975113 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.976274 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.978254 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:26.978860 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:26.978980 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:26.984332 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:26.984346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:26.984363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:26.984402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:26.988304 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.989018 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.989027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.989032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.989209 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:26.989617 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:26.989624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:26.989628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:26.990077 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:26.990347 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:26.990635 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 02:52:26.990869 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:52:26.991909 23378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:52:27.080238 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 23246
I20260621 02:52:27.096841 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 23389
Jun 21 02:52:27 dist-test-slave-bc52 krb5kdc[22551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:52:27.115295 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:52:27.097719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.11.109.3:36189)
0621 02:52:27.097912 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:52:27.097916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:52:27.097981 (+ 65us) server_negotiation.cc:407] Connection header received
0621 02:52:27.098215 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:52:27.098219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:52:27.098280 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:52:27.098376 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:52:27.100145 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:27.101052 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:27.103002 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:52:27.103453 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:52:27.103557 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:52:27.109225 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:52:27.109245 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:52:27.109261 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:52:27.109314 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:52:27.112280 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:27.113160 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:27.113167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:27.113169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:27.113550 (+ 381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:52:27.113948 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:52:27.113951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:52:27.113953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:52:27.114293 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:52:27.114531 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:52:27.114843 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 02:52:27.114997 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:52:27.116087 23512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:52:27.234577 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 23389
I20260621 02:52:27.249778 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 22995
I20260621 02:52:27.250917 23057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:52:27.360692 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 22995
I20260621 02:52:27.383128 11700 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 02:52:27.717483 11700 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 02:52:27.717706 11700 mini_ranger.cc:66] Stopping Ranger...
I20260621 02:52:28.054905 11700 mini_ranger.cc:68] Stopped Ranger
2026-06-21 02:52:28.055 UTC [22581] LOG: received smart shutdown request
2026-06-21 02:52:28.058 UTC [22581] LOG: background worker "logical replication launcher" (PID 22590) exited with exit code 1
2026-06-21 02:52:28.059 UTC [22585] LOG: shutting down
2026-06-21 02:52:28.215 UTC [22581] LOG: database system is shut down
2026-06-21T02:52:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46877 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:52:28 dist-test-slave-bc52 krb5kdc[23575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:52:28 dist-test-slave-bc52 krb5kdc[23575](info): set up 2 sockets
Jun 21 02:52:28 dist-test-slave-bc52 krb5kdc[23575](info): commencing operation
krb5kdc: starting...
W20260621 02:52:29.441639 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:52:29 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:52:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/rangeradmin_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/rangeradmin_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/rangerlookup_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/rangerlookup_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
I20260621 02:52:29.729151 11700 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSYTxIs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/postgres -l logfile start
2026-06-21 02:52:32.100 UTC [23605] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 02:52:32.101 UTC [23605] LOG: listening on IPv4 address "127.11.109.20", port 41393
2026-06-21 02:52:32.104 UTC [23605] LOG: listening on Unix socket "/tmp/.s.PGSQL.41393"
2026-06-21 02:52:32.123 UTC [23608] LOG: database system was shut down at 2026-06-21 02:52:30 UTC
2026-06-21 02:52:32.126 UTC [23605] LOG: database system is ready to accept connections
I20260621 02:52:33.081431 11700 mini_postgres.cc:96] Postgres bound to 41393
2026-06-21 02:52:33.097 UTC [23616] FATAL: database "slave" does not exist
127.11.109.20:41393 - accepting connections
I20260621 02:52:33.098213 11700 mini_ranger.cc:161] Starting Ranger...
I20260621 02:52:33.122670 11700 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 02:52:33.571827 11700 mini_ranger.cc:87] Created ranger Postgres database
I20260621 02:52:33.571959 11700 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-admin
2026-06-21 02:52:33,874 [I] DB FLAVOR :POSTGRES
2026-06-21 02:52:33,874 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:52:33,874 [I] Checking connection..
2026-06-21 02:52:33,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 02:52:34,119 [I] Checking connection passed.
2026-06-21 02:52:34,120 [I] --------- Verifying version history table ---------
2026-06-21 02:52:34,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:52:34,345 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:52:34,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:52:34,573 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:52:34,573 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 02:52:34,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 02:52:34.825 UTC [23680] WARNING: there is no transaction in progress
2026-06-21 02:52:34,831 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 02:52:34,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:52:35,090 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 02:52:35,090 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:52:35,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 02:52:35,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 02:52:35,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bc52.c.gcp-upstream.internal','N') ;"
2026-06-21 02:52:35,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:52:36,036 [I] Table x_portal_user does not exist in database ranger
2026-06-21 02:52:36,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:52:36,256 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 02:52:36,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:52:36,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:52:36,732 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 02:52:36,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 02:52:37.675 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.692 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.708 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.723 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.740 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.757 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.813 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.819 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.832 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.842 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.853 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.865 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.874 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.880 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.886 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:37.892 UTC [23788] WARNING: there is no transaction in progress
2026-06-21 02:52:38,482 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 02:52:38,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 02:52:38,708 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 02:52:38,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:52:38,961 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 02:52:38,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:52:39,220 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 02:52:39,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:52:39,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:52:39,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bc52.c.gcp-upstream.internal';"
2026-06-21 02:52:39,937 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 02:52:39,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 02:52:40,172 [I] DB_PATCHES have already been applied
I20260621 02:52:40.177937 11700 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSYTxIs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 02:52:40.178022 11700 mini_ranger.cc:193] Using host: 127.11.109.20
I20260621 02:52:40.192229 11700 mini_ranger.cc:227] Ranger admin URL: http://127.11.109.20:55507
Jun 21, 2026 2:52:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:52:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:52:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 2:52:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 2:52:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 2:52:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 2:52:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 02:52:41.306766 11700 mini_ranger.cc:160] Time spent starting Ranger: real 8.209s user 0.001s sys 0.034s
Jun 21, 2026 2:53:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 2:53:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 2:53:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 2:53:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 2:53:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 02:53:01.769011 11700 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/rangerkms_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/rangerkms_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/HTTP_127.11.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 02:53:01 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010381, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 02:53:01.931720 11700 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 02:53:01.950158 11700 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 02:53:02.707454 11700 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 02:53:02.707543 11700 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms
I20260621 02:53:02.707571 11700 mini_ranger_kms.cc:227] Using postgres at 127.11.109.20:41393
2026-06-21 02:53:03,050 [I] DB FLAVOR :POSTGRES
2026-06-21 02:53:03,051 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:53:03,051 [I] Checking connection
2026-06-21 02:53:03,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 02:53:03,274 [I] connection success
2026-06-21 02:53:03,274 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 02:53:03,274 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 02:53:03,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 02:53:03,511 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 02:53:03,511 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:53:03,511 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 02:53:03,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.109.20:41393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 02:53:03,755 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 02:53:04.189106 11700 mini_ranger_kms.cc:310] Created kms service
I20260621 02:53:04.379666 11700 mini_ranger_kms.cc:326] Created kudu user
I20260621 02:53:04.424239 11700 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 02:53:04 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010381, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:04.820912 11700 mini_ranger_kms.cc:384] Added ranger policy
I20260621 02:53:04.821305 11700 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSYTxIs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSYTxIs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 02:53:04.821370 11700 mini_ranger_kms.cc:240] Using host: 127.11.109.20
I20260621 02:53:04.835703 11700 mini_ranger_kms.cc:276] Ranger KMS PID: 23968
I20260621 02:53:04.835826 11700 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.109.20:44247
Jun 21, 2026 2:53:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:53:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 2:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 2:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 2:53:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 02:53:07.105144 11700 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.173s user 0.068s sys 0.287s
I20260621 02:53:07.105406 11700 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 02:53:07.105574 11700 mini_ranger_kms.cc:401] 127.11.109.20:44247/kms/v1/keys
Jun 21 02:53:07 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010387, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 02:53:07 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010387, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.109.20@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
Jun 21 02:53:08 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010387, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.109.20@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
Jun 21 02:53:10 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:53:10.350951 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46681
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:46681
--ranger_config_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:10.677800 24019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:10.678054 24019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:10.678197 24019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:10.685256 24019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:10.685345 24019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:10.685437 24019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:10.685528 24019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:10.685608 24019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:10.695000 24019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.109.20:44247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:46681
--ranger_config_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:10.696161 24019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:10.697510 24019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:10.709416 24025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:10.709444 24024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:10.710271 24027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:10.710749 24019 server_base.cc:1034] running on GCE node
I20260621 02:53:10.711650 24019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:10.713359 24019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:10.714608 24019 hybrid_clock.cc:648] HybridClock initialized: now 1782010390714551 us; error 55 us; skew 500 ppm
I20260621 02:53:10.714993 24019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:10 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:10.718586 24019 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:10.720163 24019 webserver.cc:460] Webserver started at http://127.11.109.62:43243/ using document root <none> and password file <none>
I20260621 02:53:10.720878 24019 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:10.721010 24019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:10.721390 24019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:53:10 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:10.783743 24019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "471e2c119ca544a3821c9ff2c61ef165"
format_stamp: "Formatted at 2026-06-21 02:53:10 on dist-test-slave-bc52"
server_key: "849b06a85365b72510a6ca928573dc06"
server_key_iv: "f340f414f8364fda4364420a7bbb73a2"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:10.784582 24019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "471e2c119ca544a3821c9ff2c61ef165"
format_stamp: "Formatted at 2026-06-21 02:53:10 on dist-test-slave-bc52"
server_key: "849b06a85365b72510a6ca928573dc06"
server_key_iv: "f340f414f8364fda4364420a7bbb73a2"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:10.789928 24019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:53:10.805420 24034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:10.806908 24019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:53:10.807075 24019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "471e2c119ca544a3821c9ff2c61ef165"
format_stamp: "Formatted at 2026-06-21 02:53:10 on dist-test-slave-bc52"
server_key: "849b06a85365b72510a6ca928573dc06"
server_key_iv: "f340f414f8364fda4364420a7bbb73a2"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:10.807283 24019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:10.836004 24019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:10.840103 24019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:10.840442 24019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:10.859521 24019 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:46681
I20260621 02:53:10.859529 24085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:46681 every 8 connection(s)
I20260621 02:53:10.860837 24019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:10.863255 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24019
I20260621 02:53:10.863471 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:53:10.864154 11700 external_mini_cluster.cc:1428] Setting key aeb12c82794f9d0f3a8ce0b8af59f62c
I20260621 02:53:10.864884 24086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:10.879382 24086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Bootstrap starting.
Jun 21 02:53:10 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:10.883515 24086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:10.884815 24086 log.cc:824] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:10.888127 24086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: No bootstrap required, opened a new log
I20260621 02:53:10.889299 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:10.866276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.11.109.62:46681)
0621 02:53:10.867075 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:10.867089 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:10.867137 (+ 48us) server_negotiation.cc:407] Connection header received
0621 02:53:10.869734 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:10.869763 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:10.870261 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:10.870679 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:10.871821 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:10.873817 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:10.875026 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:10.876101 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:10.876274 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:10.881019 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:10.881030 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:10.881053 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:10.881114 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:10.884964 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:10.885767 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:10.885772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:10.885783 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:10.885936 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:10.886388 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:10.886392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:10.886395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:10.887013 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:10.887302 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:10.887625 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 02:53:10.887988 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":330,"threads_started":1}
I20260621 02:53:10.896260 24086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:10.896625 24086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:10.896744 24086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471e2c119ca544a3821c9ff2c61ef165, State: Initialized, Role: FOLLOWER
I20260621 02:53:10.897411 24086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:10.897644 24086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:10.897740 24086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:10.897851 24086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:10.900084 24086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:10.900576 24086 leader_election.cc:302] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471e2c119ca544a3821c9ff2c61ef165; no voters:
I20260621 02:53:10.901111 24086 leader_election.cc:288] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:53:10.901276 24091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:10.901928 24091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 LEADER]: Becoming Leader. State: Replica: 471e2c119ca544a3821c9ff2c61ef165, State: Running, Role: LEADER
I20260621 02:53:10.902426 24091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:10.903108 24086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:10.905773 24093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471e2c119ca544a3821c9ff2c61ef165. Latest consensus state: current_term: 1 leader_uuid: "471e2c119ca544a3821c9ff2c61ef165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } } }
I20260621 02:53:10.905872 24092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "471e2c119ca544a3821c9ff2c61ef165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } } }
I20260621 02:53:10.906195 24093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:10.906248 24092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:10.908001 24096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:10.913404 24096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:10.914139 24086 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:53:10.929898 24096 catalog_manager.cc:1347] Generated new cluster ID: a21a126fd367485ab866aa4e578b8caf
I20260621 02:53:10.930035 24096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:10.950573 24096 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:53:10.952500 24096 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:10.963410 24096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Generated new TSK 0
I20260621 02:53:10.964443 24096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:53:11.661006 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46681
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:53:11 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010391, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 02:53:12.059439 24131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:12.059687 24131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:12.059829 24131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:12.068451 24131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:12.068550 24131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:12.068712 24131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:53:12.080729 24131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.109.20:44247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:12.082077 24131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:12.083314 24131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:12.103027 24141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:12.105044 24142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:12.117146 24131 server_base.cc:1034] running on GCE node
W20260621 02:53:12.118054 24145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:12.118813 24131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:12.119894 24131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:12.121161 24131 hybrid_clock.cc:648] HybridClock initialized: now 1782010392121098 us; error 59 us; skew 500 ppm
I20260621 02:53:12.121583 24131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:12 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010392, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:12.125110 24131 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:53:12.130568 24131 webserver.cc:460] Webserver started at http://127.11.109.1:42493/ using document root <none> and password file <none>
I20260621 02:53:12.131321 24131 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:12.131459 24131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:12.131783 24131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:53:12 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010392, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:12.153451 24131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "0305e76492864f4ba9c0d55b1643114c"
format_stamp: "Formatted at 2026-06-21 02:53:12 on dist-test-slave-bc52"
server_key: "1ea71b5ae9fb7ab426ed84b0cf60da4b"
server_key_iv: "597513601e6057a2b7f544591c51c07a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:12.156544 24131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "0305e76492864f4ba9c0d55b1643114c"
format_stamp: "Formatted at 2026-06-21 02:53:12 on dist-test-slave-bc52"
server_key: "1ea71b5ae9fb7ab426ed84b0cf60da4b"
server_key_iv: "597513601e6057a2b7f544591c51c07a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:12.163568 24131 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 02:53:12.189829 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:12.191721 24131 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260621 02:53:12.191910 24131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "0305e76492864f4ba9c0d55b1643114c"
format_stamp: "Formatted at 2026-06-21 02:53:12 on dist-test-slave-bc52"
server_key: "1ea71b5ae9fb7ab426ed84b0cf60da4b"
server_key_iv: "597513601e6057a2b7f544591c51c07a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:12.192166 24131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:12.225401 24131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:12.230576 24131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:12.230964 24131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:12.241734 24131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:12.254005 24131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:12.254174 24131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:12.254375 24131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:12.254488 24131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:53:12 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010391, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:12.659680 24131 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:34387
I20260621 02:53:12.659703 24263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:34387 every 8 connection(s)
I20260621 02:53:12.661067 24131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:53:12.661827 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24131
I20260621 02:53:12.662065 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:53:12.663060 11700 external_mini_cluster.cc:1428] Setting key 348d3170c3d1509e0cc7ae9ae54af061
Jun 21 02:53:12 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010392, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
I20260621 02:53:12.709718 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:12.680302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:33491 (local address 127.11.109.62:46681)
0621 02:53:12.685538 (+ 5236us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:12.685544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:12.689274 (+ 3730us) server_negotiation.cc:407] Connection header received
0621 02:53:12.689350 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:12.689358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:12.689458 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:12.689566 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:12.691695 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:12.693412 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:12.696053 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:12.696809 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:12.696921 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:12.702052 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:12.702060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:12.702066 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:12.702108 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:12.705963 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:12.706876 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:12.706881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:12.706885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:12.707039 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:12.707562 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:12.707567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:12.707569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:12.707917 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:12.708036 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:12.709073 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0621 02:53:12.709322 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5119,"thread_start_us":5008,"threads_started":1}
I20260621 02:53:12.712342 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:12.680831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46681 (local address 127.11.109.1:33491)
0621 02:53:12.686192 (+ 5361us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:12.686245 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:12.687947 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:12.689995 (+ 2048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:12.690010 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:12.690698 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:12.691488 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:12.691515 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:12.693622 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:12.693631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:12.695897 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:12.695913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:12.697311 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:12.697320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:12.697651 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:12.698546 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:12.698581 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:12.701763 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:12.706127 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:12.706136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:12.706155 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:12.706734 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:12.707182 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:12.707187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:12.707191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:12.707430 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:12.708112 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:12.708120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:12.708638 (+ 518us) client_negotiation.cc:769] Sending connection context
0621 02:53:12.710235 (+ 1597us) client_negotiation.cc:240] Negotiation successful
0621 02:53:12.710787 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5101,"thread_start_us":4945,"threads_started":1}
I20260621 02:53:12.714385 24264 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46681
I20260621 02:53:12.714691 24264 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:12.715461 24264 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:12.717850 24051 ts_manager.cc:194] Registered new tserver with Master: 0305e76492864f4ba9c0d55b1643114c (127.11.109.1:34387)
I20260621 02:53:12.719728 24051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:33491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:53:12.746948 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46681
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:53:12 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010391, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
W20260621 02:53:13.181860 24271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:13.182255 24271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:13.182444 24271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:13.189946 24271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:13.190093 24271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:13.190258 24271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:53:13.200141 24271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.109.20:44247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:13.201423 24271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:13.202710 24271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:13.216758 24284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:13.216825 24285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:13.217952 24287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:13.218034 24271 server_base.cc:1034] running on GCE node
I20260621 02:53:13.218629 24271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:13.219583 24271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:13.220824 24271 hybrid_clock.cc:648] HybridClock initialized: now 1782010393220779 us; error 58 us; skew 500 ppm
I20260621 02:53:13.221127 24271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010393, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:13.224908 24271 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:53:13.226598 24271 webserver.cc:460] Webserver started at http://127.11.109.2:43327/ using document root <none> and password file <none>
I20260621 02:53:13.227420 24271 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:13.227550 24271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:13.227864 24271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010393, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:13.245364 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "532f8c22bcd047aeb27f950316f6d92a"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "560c93b8ce8aef49726025ce1490e4ba"
server_key_iv: "262694cb97e609f3d962430f1ac07e6a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:13.246599 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "532f8c22bcd047aeb27f950316f6d92a"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "560c93b8ce8aef49726025ce1490e4ba"
server_key_iv: "262694cb97e609f3d962430f1ac07e6a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:13.253489 24271 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 02:53:13.272146 24294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:13.274050 24271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:53:13.274302 24271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "532f8c22bcd047aeb27f950316f6d92a"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "560c93b8ce8aef49726025ce1490e4ba"
server_key_iv: "262694cb97e609f3d962430f1ac07e6a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:13.274605 24271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:13.309598 24271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:13.314881 24271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:13.315301 24271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:13.316545 24271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:13.318306 24271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:13.318441 24271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260621 02:53:13.318617 24271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:13.318715 24271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:13.355054 24271 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:39305
I20260621 02:53:13.355176 24406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:39305 every 8 connection(s)
I20260621 02:53:13.356791 24271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:53:13.365514 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24271
I20260621 02:53:13.365787 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:53:13.366354 11700 external_mini_cluster.cc:1428] Setting key 7c26b992e4a0c563584a0fe43ebace90
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010393, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
I20260621 02:53:13.397938 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:13.360440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46681 (local address 127.11.109.2:48681)
0621 02:53:13.369002 (+ 8562us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:13.369053 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:13.371693 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:13.372750 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:13.372764 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:13.373467 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:13.374292 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:13.374325 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.375856 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.375863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:13.378070 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:13.378087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.379418 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.379427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:13.379890 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:13.380635 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:13.380668 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:13.383640 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:13.393156 (+ 9516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:13.393164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:13.393228 (+ 64us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:13.393707 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:13.394164 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:13.394168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:13.394173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:13.394372 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:13.394984 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:13.394992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:13.395413 (+ 421us) client_negotiation.cc:769] Sending connection context
0621 02:53:13.395710 (+ 297us) client_negotiation.cc:240] Negotiation successful
0621 02:53:13.396269 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8270,"thread_start_us":8017,"threads_started":1}
I20260621 02:53:13.399891 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:13.359655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:48681 (local address 127.11.109.62:46681)
0621 02:53:13.368449 (+ 8794us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:13.368455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:13.369554 (+ 1099us) server_negotiation.cc:407] Connection header received
0621 02:53:13.372093 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:13.372113 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:13.372203 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:13.372306 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:13.374492 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.375682 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.378221 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.379009 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.379134 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:13.383887 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:13.383897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:13.383905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:13.383950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:13.387877 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:13.393840 (+ 5963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:13.393848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:13.393855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:13.394031 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:13.394495 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:13.394500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:13.394504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:13.394862 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:13.395015 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:13.399134 (+ 4119us) server_negotiation.cc:299] Negotiation successful
0621 02:53:13.399423 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8664,"thread_start_us":9769,"threads_started":1}
I20260621 02:53:13.401436 24407 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46681
I20260621 02:53:13.401762 24407 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:13.402609 24407 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:13.404296 24051 ts_manager.cc:194] Registered new tserver with Master: 532f8c22bcd047aeb27f950316f6d92a (127.11.109.2:39305)
I20260621 02:53:13.405071 24051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:48681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:53:13.426347 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46681
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:13.723259 24264 heartbeater.cc:503] Master 127.11.109.62:46681 was elected leader, sending a full tablet report...
W20260621 02:53:13.752995 24413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:13.753346 24413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:13.753520 24413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:13.760533 24413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:13.760663 24413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:13.760843 24413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:53:13.770525 24413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.109.20:44247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:13.771806 24413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:13.773097 24413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:13.798803 24418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:13.798832 24419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:13.799656 24421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:13.800004 24413 server_base.cc:1034] running on GCE node
I20260621 02:53:13.800625 24413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:13.801679 24413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:13.802901 24413 hybrid_clock.cc:648] HybridClock initialized: now 1782010393802852 us; error 43 us; skew 500 ppm
I20260621 02:53:13.803282 24413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010393, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:13.809603 24413 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:53:13.811303 24413 webserver.cc:460] Webserver started at http://127.11.109.3:45213/ using document root <none> and password file <none>
I20260621 02:53:13.812170 24413 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:13.812297 24413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:13.812609 24413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010393, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:13.829639 24413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "71c102b00a2a4b4fb556b9950b55dad6"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "46d1c8e68001d64c28d19a8e310e4026"
server_key_iv: "216b1cf22cad689dd006627cca2af699"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:13.830612 24413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "71c102b00a2a4b4fb556b9950b55dad6"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "46d1c8e68001d64c28d19a8e310e4026"
server_key_iv: "216b1cf22cad689dd006627cca2af699"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:13.837303 24413 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 02:53:13.854066 24428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:13.855715 24413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:53:13.855880 24413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "71c102b00a2a4b4fb556b9950b55dad6"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "46d1c8e68001d64c28d19a8e310e4026"
server_key_iv: "216b1cf22cad689dd006627cca2af699"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:13.856097 24413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:13.885243 24413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:13.890455 24413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:13.890822 24413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:13.900485 24413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:13.902215 24413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:13.902320 24413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:13.902451 24413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:13.902534 24413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:13.933248 24413 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35167
I20260621 02:53:13.933285 24540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35167 every 8 connection(s)
I20260621 02:53:13.934506 24413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:53:13.937973 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24413
I20260621 02:53:13.938176 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:53:13.938777 11700 external_mini_cluster.cc:1428] Setting key 6cfbe2ccaa2bfc6602fbb0a41b246a0c
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010393, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:13.953825 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:13.937046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:39547 (local address 127.11.109.62:46681)
0621 02:53:13.937578 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:13.937584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:13.938727 (+ 1143us) server_negotiation.cc:407] Connection header received
0621 02:53:13.941051 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:13.941057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:13.941161 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:13.941302 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:13.942775 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.943496 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.945028 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.945519 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.945614 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:13.948537 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:13.948542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:13.948545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:13.948575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:13.950984 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:13.951563 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:13.951566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:13.951568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:13.951680 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:13.952027 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:13.952030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:13.952032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:13.952298 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:13.952439 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:13.953157 (+ 718us) server_negotiation.cc:299] Negotiation successful
0621 02:53:13.953424 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":252,"threads_started":1}
I20260621 02:53:13.954769 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:13.937709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46681 (local address 127.11.109.3:39547)
0621 02:53:13.938537 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:13.938595 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:13.940771 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:13.941502 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:13.941510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:13.942068 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:13.942629 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:13.942646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.943631 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.943634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:13.944913 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:13.944924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.945718 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.945723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:13.945915 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:13.946463 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:13.946482 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:13.948347 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:13.951095 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:13.951101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:13.951112 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:13.951473 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:13.951771 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:13.951774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:13.951776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:13.951942 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:13.952438 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:13.952443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:13.952783 (+ 340us) client_negotiation.cc:769] Sending connection context
0621 02:53:13.953088 (+ 305us) client_negotiation.cc:240] Negotiation successful
0621 02:53:13.953627 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
I20260621 02:53:13.955957 24541 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46681
I20260621 02:53:13.956195 24541 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:13.956810 24541 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:13.958127 24051 ts_manager.cc:194] Registered new tserver with Master: 71c102b00a2a4b4fb556b9950b55dad6 (127.11.109.3:35167)
I20260621 02:53:13.958932 24051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:39547
I20260621 02:53:13.966085 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:53:13.967748 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24131
Jun 21 02:53:13 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:53:13.983978 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:13.968779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.11.109.1:34387)
0621 02:53:13.969440 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:13.969448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:13.969479 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:53:13.969531 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:13.969536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:13.969715 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:13.969870 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:13.971207 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.972216 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.973732 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:13.974264 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:13.974398 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:13.978467 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:13.978477 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:13.978489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:13.978522 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:13.981279 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:13.981915 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:13.981922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:13.981924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:13.982067 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:13.982439 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:13.982443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:13.982445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:13.982742 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:13.983037 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:13.983383 (+ 346us) server_negotiation.cc:299] Negotiation successful
0621 02:53:13.983660 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":288,"threads_started":1}
I20260621 02:53:13.985163 24259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:14.066526 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24131
I20260621 02:53:14.081624 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24271
Jun 21 02:53:14 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:53:14.097427 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:14.082502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.11.109.2:39305)
0621 02:53:14.082986 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:14.082993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:14.083017 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:53:14.083085 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:14.083092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:14.083330 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:14.083453 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:14.084582 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:14.085602 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:14.086872 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:14.087410 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:14.087506 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:14.091574 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:14.091584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:14.091604 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:14.091634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:14.094450 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:14.095200 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:14.095208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:14.095211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:14.095337 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:14.095711 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:14.095718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:14.095721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:14.096049 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:14.096374 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:14.096706 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 02:53:14.096967 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":237,"threads_started":1}
I20260621 02:53:14.098732 24402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:14.178656 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24271
I20260621 02:53:14.198439 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24413
Jun 21 02:53:14 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:53:14.215020 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:14.199491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.11.109.3:35167)
0621 02:53:14.200082 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:14.200089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:14.200124 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:53:14.200182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:14.200188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:14.200392 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:14.200560 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:14.201961 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:14.202982 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:14.204319 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:14.204942 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:14.205087 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:14.209246 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:14.209260 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:14.209278 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:14.209320 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:14.212194 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:14.212903 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:14.212911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:14.212913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:14.213028 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:14.213530 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:14.213539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:14.213541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:14.213847 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:14.214194 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:14.214425 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 02:53:14.214676 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
I20260621 02:53:14.216202 24536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:14.296590 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24413
I20260621 02:53:14.312381 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24019
I20260621 02:53:14.313623 24081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:14.417987 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24019
I20260621 02:53:14.439514 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46681
--webserver_interface=127.11.109.62
--webserver_port=43243
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:46681
--ranger_config_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:14.751063 24553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:14.751313 24553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:14.751497 24553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:14.758348 24553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:14.758442 24553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:14.758520 24553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:14.758612 24553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:14.758713 24553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:14.768486 24553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:46681
--ranger_config_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=43243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:14.769766 24553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:14.771039 24553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:14.782711 24558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:14.782873 24561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:14.782732 24559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:14.783454 24553 server_base.cc:1034] running on GCE node
I20260621 02:53:14.784152 24553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:14.785494 24553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:14.786718 24553 hybrid_clock.cc:648] HybridClock initialized: now 1782010394786671 us; error 56 us; skew 500 ppm
I20260621 02:53:14.787048 24553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:14 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010394, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:14.790153 24553 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:14.791450 24553 webserver.cc:460] Webserver started at http://127.11.109.62:43243/ using document root <none> and password file <none>
I20260621 02:53:14.792016 24553 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:14.792124 24553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:53:14 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010394, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:14.838976 24553 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:53:14.842144 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:14.843650 24553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:53:14.843847 24553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "471e2c119ca544a3821c9ff2c61ef165"
format_stamp: "Formatted at 2026-06-21 02:53:10 on dist-test-slave-bc52"
server_key: "849b06a85365b72510a6ca928573dc06"
server_key_iv: "f340f414f8364fda4364420a7bbb73a2"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:14.844744 24553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:14.864706 24553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:14.868399 24553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:14.868676 24553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:14.887439 24553 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:46681
I20260621 02:53:14.887458 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:46681 every 8 connection(s)
I20260621 02:53:14.888929 24553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:14.889789 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24553
I20260621 02:53:14.890764 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:34387
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:46681
--webserver_port=42493
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:14.896065 24621 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:53:14.900877 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Bootstrap starting.
I20260621 02:53:14.928661 24621 log.cc:824] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:14.939348 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:53:14.939965 24621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Bootstrap complete.
I20260621 02:53:14.953331 24621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:14.954715 24621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471e2c119ca544a3821c9ff2c61ef165, State: Initialized, Role: FOLLOWER
I20260621 02:53:14.955546 24621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:14.955889 24621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:14.956039 24621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:14.956243 24621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:53:14.962204 24621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:14.962785 24621 leader_election.cc:302] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471e2c119ca544a3821c9ff2c61ef165; no voters:
I20260621 02:53:14.963428 24621 leader_election.cc:288] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:53:14.963570 24625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:53:14.964989 24625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [term 2 LEADER]: Becoming Leader. State: Replica: 471e2c119ca544a3821c9ff2c61ef165, State: Running, Role: LEADER
I20260621 02:53:14.965431 24621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:14.965560 24625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } }
I20260621 02:53:14.967803 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471e2c119ca544a3821c9ff2c61ef165. Latest consensus state: current_term: 2 leader_uuid: "471e2c119ca544a3821c9ff2c61ef165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } } }
I20260621 02:53:14.968137 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:14.968652 24626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "471e2c119ca544a3821c9ff2c61ef165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471e2c119ca544a3821c9ff2c61ef165" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46681 } } }
I20260621 02:53:14.969020 24626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:14.970930 24631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:14.976356 24631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:14.977747 24631 catalog_manager.cc:1259] Loaded cluster ID: a21a126fd367485ab866aa4e578b8caf
I20260621 02:53:14.977913 24631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:14.982043 24631 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:14.983515 24631 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Loaded TSK: 0
I20260621 02:53:14.985019 24631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:53:14.988126 24621 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260621 02:53:15.313197 24623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:15.313663 24623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:15.313862 24623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:15.324208 24623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:15.324364 24623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:15.324532 24623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:53:15.338564 24623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:34387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=42493
--tserver_master_addrs=127.11.109.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:15.340230 24623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:15.341715 24623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:15.386974 24659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:15.388301 24658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:15.388595 24661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:15.389676 24623 server_base.cc:1034] running on GCE node
I20260621 02:53:15.390662 24623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:15.393718 24623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:15.395035 24623 hybrid_clock.cc:648] HybridClock initialized: now 1782010395394959 us; error 65 us; skew 500 ppm
I20260621 02:53:15.395427 24623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:15 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010395, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:15.399853 24623 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:53:15.401851 24623 webserver.cc:460] Webserver started at http://127.11.109.1:42493/ using document root <none> and password file <none>
I20260621 02:53:15.402786 24623 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:15.403023 24623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:53:15 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010395, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:15.436586 24623 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:53:15.440261 24669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:15.441991 24623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260621 02:53:15.442240 24623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "0305e76492864f4ba9c0d55b1643114c"
format_stamp: "Formatted at 2026-06-21 02:53:12 on dist-test-slave-bc52"
server_key: "1ea71b5ae9fb7ab426ed84b0cf60da4b"
server_key_iv: "597513601e6057a2b7f544591c51c07a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:15.443651 24623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:15.478193 24623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:15.483641 24623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:15.484455 24623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:15.505438 24623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:15.510035 24623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:15.510239 24623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:15.510439 24623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:15.510573 24623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:15.582286 24623 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:34387
I20260621 02:53:15.582332 24782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:34387 every 8 connection(s)
I20260621 02:53:15.583827 24623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:53:15.591485 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24623
I20260621 02:53:15.592177 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:39305
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:46681
--webserver_port=43327
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:53:15 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010395, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:15.632174 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:15.587828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46681 (local address 127.11.109.1:56269)
0621 02:53:15.608129 (+ 20301us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:15.608182 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:15.610047 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:15.611526 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:15.611539 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:15.612081 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:15.612627 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:15.612654 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:15.614296 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:15.614308 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:15.615515 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:15.615526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:15.616564 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:15.616579 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:15.616851 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:15.617961 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:15.618000 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:15.621131 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:15.626620 (+ 5489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:15.626629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:15.626644 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:15.627224 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:15.627808 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:15.627814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:15.627818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:15.628105 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:15.628971 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:15.628979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:15.629832 (+ 853us) client_negotiation.cc:769] Sending connection context
0621 02:53:15.630180 (+ 348us) client_negotiation.cc:240] Negotiation successful
0621 02:53:15.630747 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19987,"thread_start_us":19740,"threads_started":1}
I20260621 02:53:15.632503 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:15.589129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:56269 (local address 127.11.109.62:46681)
0621 02:53:15.607450 (+ 18321us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:15.607479 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:15.609284 (+ 1805us) server_negotiation.cc:407] Connection header received
0621 02:53:15.610634 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:15.610648 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:15.611007 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:15.611390 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:15.612846 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:15.614135 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:15.615669 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:15.616324 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:15.616530 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:15.621457 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:15.621467 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:15.621487 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:15.621536 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:15.626451 (+ 4915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:15.627387 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:15.627397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:15.627405 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:15.627634 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:15.628250 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:15.628255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:15.628259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:15.628827 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:15.629030 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:15.630871 (+ 1841us) server_negotiation.cc:299] Negotiation successful
0621 02:53:15.631308 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18004,"thread_start_us":17749,"threads_started":1}
I20260621 02:53:15.636449 24783 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46681
I20260621 02:53:15.636775 24783 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:15.637704 24783 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
W20260621 02:53:15.640969 24783 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:46681 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:53:15.641214 24783 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:15.641510 24783 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:15.642899 24783 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:15.643200 24783 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
W20260621 02:53:15.644276 24783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:53:15.998677 24787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:15.998960 24787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:15.999104 24787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:16.006619 24787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:16.006719 24787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:16.006883 24787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:53:16.019153 24787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:39305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=43327
--tserver_master_addrs=127.11.109.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:16.020591 24787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:16.022141 24787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:16.039108 24802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:16.044170 24801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:16.044696 24787 server_base.cc:1034] running on GCE node
W20260621 02:53:16.044983 24804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:16.045804 24787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:16.047019 24787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:16.048283 24787 hybrid_clock.cc:648] HybridClock initialized: now 1782010396048233 us; error 38 us; skew 500 ppm
I20260621 02:53:16.048661 24787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:16.052909 24787 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:53:16.056607 24787 webserver.cc:460] Webserver started at http://127.11.109.2:43327/ using document root <none> and password file <none>
I20260621 02:53:16.057521 24787 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:16.057698 24787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:16.080353 24787 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:53:16.083855 24812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:16.085706 24787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260621 02:53:16.085922 24787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "532f8c22bcd047aeb27f950316f6d92a"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "560c93b8ce8aef49726025ce1490e4ba"
server_key_iv: "262694cb97e609f3d962430f1ac07e6a"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:16.087294 24787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:16.155042 24787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:16.159516 24787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:16.159817 24787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:16.160889 24787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:16.162379 24787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:16.162499 24787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:16.162647 24787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:16.162739 24787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:16.219624 24787 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:39305
I20260621 02:53:16.219707 24926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:39305 every 8 connection(s)
I20260621 02:53:16.221575 24787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:53:16.228685 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24787
I20260621 02:53:16.229501 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.109.20:44247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:35167
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:46681
--webserver_port=45213
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:16.273942 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:16.244505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:53009 (local address 127.11.109.62:46681)
0621 02:53:16.249920 (+ 5415us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:16.249927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:16.254795 (+ 4868us) server_negotiation.cc:407] Connection header received
0621 02:53:16.256301 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:16.256307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:16.256401 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:16.256500 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:16.258696 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:16.259869 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:16.262307 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:16.262974 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:16.263095 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:16.267781 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:16.267788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:16.267793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:16.267834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:16.270784 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:16.271480 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:16.271493 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:16.271495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:16.271635 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:16.272083 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:16.272088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:16.272091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:16.272379 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:16.272543 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:16.273362 (+ 819us) server_negotiation.cc:299] Negotiation successful
0621 02:53:16.273572 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5281,"thread_start_us":4841,"threads_started":1}
I20260621 02:53:16.274961 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:16.243911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46681 (local address 127.11.109.2:53009)
0621 02:53:16.250347 (+ 6436us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:16.250405 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:16.255806 (+ 5401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:16.256972 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:16.256985 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:16.257658 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:16.258487 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:16.258513 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:16.260069 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:16.260074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:16.262131 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:16.262146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:16.263479 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:16.263485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:16.263766 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:16.264564 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:16.264595 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:16.267518 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:16.270922 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:16.270931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:16.270949 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:16.271357 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:16.271755 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:16.271760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:16.271764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:16.271964 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:16.272531 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:16.272538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:16.272969 (+ 431us) client_negotiation.cc:769] Sending connection context
0621 02:53:16.273256 (+ 287us) client_negotiation.cc:240] Negotiation successful
0621 02:53:16.273793 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6032,"thread_start_us":5503,"threads_started":1}
I20260621 02:53:16.276705 24927 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46681
I20260621 02:53:16.277030 24927 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:16.277768 24927 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:16.279577 24586 ts_manager.cc:194] Registered new tserver with Master: 532f8c22bcd047aeb27f950316f6d92a (127.11.109.2:39305)
I20260621 02:53:16.281111 24586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:53009
I20260621 02:53:16.646338 24783 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:16.646791 24783 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:16.648181 24586 ts_manager.cc:194] Registered new tserver with Master: 0305e76492864f4ba9c0d55b1643114c (127.11.109.1:34387)
I20260621 02:53:16.649554 24586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:56269
W20260621 02:53:16.649796 24929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:16.650216 24929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:16.650389 24929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:16.662120 24929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:16.662329 24929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:16.662573 24929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:53:16.675197 24929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.109.20:44247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:35167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=45213
--tserver_master_addrs=127.11.109.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:16.676503 24929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:16.678265 24929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
W20260621 02:53:16.697501 24937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:16.701508 24938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:16.747591 24940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:16.750279 24929 server_base.cc:1034] running on GCE node
I20260621 02:53:16.751132 24929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:16.766105 24929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:16.770206 24929 hybrid_clock.cc:648] HybridClock initialized: now 1782010396770116 us; error 81 us; skew 500 ppm
I20260621 02:53:16.770673 24929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:16.775221 24929 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:53:16.803331 24929 webserver.cc:460] Webserver started at http://127.11.109.3:45213/ using document root <none> and password file <none>
I20260621 02:53:16.804471 24929 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:16.804719 24929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:53:16 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:16.857775 24929 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.000s sys 0.006s
I20260621 02:53:16.867219 24948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:16.869104 24929 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.007s
I20260621 02:53:16.869382 24929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "71c102b00a2a4b4fb556b9950b55dad6"
format_stamp: "Formatted at 2026-06-21 02:53:13 on dist-test-slave-bc52"
server_key: "46d1c8e68001d64c28d19a8e310e4026"
server_key_iv: "216b1cf22cad689dd006627cca2af699"
server_key_version: "kuduclusterkey@0"
I20260621 02:53:16.870929 24929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:16.964695 24929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:16.970052 24929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:16.970413 24929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:16.977756 24929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:16.986119 24929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:16.986289 24929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:16.986433 24929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:16.986538 24929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:53:17 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for HTTP/127.11.109.20@KRBTEST.COM
I20260621 02:53:17.277405 24929 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35167
I20260621 02:53:17.277486 25061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35167 every 8 connection(s)
I20260621 02:53:17.279338 24929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:53:17.284516 24927 heartbeater.cc:503] Master 127.11.109.62:46681 was elected leader, sending a full tablet report...
I20260621 02:53:17.288395 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 24929
I20260621 02:53:17.313364 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.289763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.11.109.62:46681)
0621 02:53:17.292523 (+ 2760us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.292529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.292548 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:53:17.292607 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.292613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.292701 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.292816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:17.297901 (+ 5085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.299065 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.301039 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.301707 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.301825 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.305714 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:17.305720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:17.305725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:17.305769 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:17.309481 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.310414 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.310419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.310423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.310565 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.311028 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.311034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.311038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.311363 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:17.311593 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.312757 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.312967 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2653,"thread_start_us":2948,"threads_started":1}
Jun 21 02:53:17 dist-test-slave-bc52 krb5kdc[23575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:17.323336 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.296365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:48039 (local address 127.11.109.62:46681)
0621 02:53:17.305024 (+ 8659us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.305029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.305046 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:53:17.305091 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.305095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.305190 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.305285 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:17.308053 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.309363 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.311719 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.312366 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.312455 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.317246 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:17.317253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:17.317258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:17.317295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:17.320369 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.321036 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.321041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.321045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.321216 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.321609 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.321614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.321618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.321933 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:17.322061 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.322729 (+ 668us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.322943 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8551,"thread_start_us":8423,"threads_started":1}
I20260621 02:53:17.324035 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.295743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:46681 (local address 127.11.109.3:48039)
0621 02:53:17.301409 (+ 5666us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:17.301454 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:17.304199 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:17.305818 (+ 1619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:17.305831 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:17.306584 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:17.307811 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.307838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.309647 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.309653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.311548 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.311563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.312810 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.312818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.313115 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.313833 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:17.313869 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:17.316948 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:17.320496 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:17.320502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:17.320514 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:17.320919 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:17.321323 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:17.321326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:17.321328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:17.321505 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:17.322072 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:17.322077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:17.322363 (+ 286us) client_negotiation.cc:769] Sending connection context
0621 02:53:17.322598 (+ 235us) client_negotiation.cc:240] Negotiation successful
0621 02:53:17.322971 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5347,"thread_start_us":5134,"threads_started":1}
I20260621 02:53:17.325493 25062 heartbeater.cc:347] Connected to a master server at 127.11.109.62:46681
I20260621 02:53:17.325788 25062 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:17.326548 25062 heartbeater.cc:511] Master 127.11.109.62:46681 requested a full tablet report, sending...
I20260621 02:53:17.328454 24586 ts_manager.cc:194] Registered new tserver with Master: 71c102b00a2a4b4fb556b9950b55dad6 (127.11.109.3:35167)
I20260621 02:53:17.329463 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:53:17.329593 24586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:48039
I20260621 02:53:17.346791 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.332776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.11.109.62:46681)
0621 02:53:17.332925 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.332931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.333414 (+ 483us) server_negotiation.cc:407] Connection header received
0621 02:53:17.333522 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.333528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.333596 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.333702 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:17.335188 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.336260 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.337675 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.338303 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.338435 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.340938 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:17.340944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:17.340949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:17.340986 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:17.344383 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.345052 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.345057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.345061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.345236 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.345607 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.345611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.345613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.345898 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:17.346059 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.346402 (+ 343us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.346572 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:53:17.351730 24586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:53:17.358811 24586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:53:17.390177 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.371901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34387 (local address 127.0.0.1:48028)
0621 02:53:17.382535 (+ 10634us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:17.382552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:17.382678 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:17.383804 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:17.383811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:17.383839 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:17.384161 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.384171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.385870 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.385876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.387422 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.387434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.389125 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.389131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.389408 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.389466 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 02:53:17.389613 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 02:53:17.389770 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10527,"thread_start_us":10655,"threads_started":1}
I20260621 02:53:17.391125 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.372834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.11.109.1:34387)
0621 02:53:17.382532 (+ 9698us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.382542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.383298 (+ 756us) server_negotiation.cc:407] Connection header received
0621 02:53:17.383419 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.383425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.383602 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.383801 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:17.384333 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.385688 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.387567 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.388575 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.388729 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.388976 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.390604 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.390790 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9521,"thread_start_us":9348,"threads_started":1}
I20260621 02:53:17.395047 24717 tablet_service.cc:1467] Processing CreateTablet for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 (DEFAULT_TABLE table=test-table [id=a311cd47f25947d7a2ea09ab4c1d12a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:17.396440 24717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:17.396792 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.372301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39305 (local address 127.0.0.1:51722)
0621 02:53:17.383157 (+ 10856us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:17.383174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:17.383281 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:17.389912 (+ 6631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:17.389917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:17.389942 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:17.390343 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.390356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.392045 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.392053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.394175 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.394193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.395989 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.395994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.396153 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.396189 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:53:17.396301 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 02:53:17.396424 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10713,"thread_start_us":8756,"threads_started":1}
I20260621 02:53:17.396827 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.373630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.11.109.2:39305)
0621 02:53:17.389124 (+ 15494us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.389137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.389198 (+ 61us) server_negotiation.cc:407] Connection header received
0621 02:53:17.389342 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.389352 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.389678 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.389914 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:17.390555 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.391906 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.394358 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.395780 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.395936 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.396151 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.396347 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.396486 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15268,"thread_start_us":16489,"threads_started":1}
I20260621 02:53:17.398326 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.371740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35167 (local address 127.0.0.1:45782)
0621 02:53:17.381574 (+ 9834us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:17.381636 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:17.381857 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:17.391102 (+ 9245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:17.391107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:17.391130 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:17.391630 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.391654 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.393623 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.393628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.395631 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.395645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.397589 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.397602 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.397765 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.397798 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:53:17.397884 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:53:17.398015 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9494,"thread_start_us":11293,"threads_started":1}
I20260621 02:53:17.399052 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.372065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.11.109.3:35167)
0621 02:53:17.390658 (+ 18593us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.390668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.390702 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:53:17.390779 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.390784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.390929 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.391115 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:17.392075 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.393475 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.395760 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.397199 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.397325 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.397528 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.398576 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.398726 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18436,"thread_start_us":18418,"threads_started":1}
I20260621 02:53:17.400658 24862 tablet_service.cc:1467] Processing CreateTablet for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 (DEFAULT_TABLE table=test-table [id=a311cd47f25947d7a2ea09ab4c1d12a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:17.401765 24862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:17.402441 24997 tablet_service.cc:1467] Processing CreateTablet for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 (DEFAULT_TABLE table=test-table [id=a311cd47f25947d7a2ea09ab4c1d12a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:17.404042 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:17.413069 25090 tablet_bootstrap.cc:492] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Bootstrap starting.
I20260621 02:53:17.414777 25091 tablet_bootstrap.cc:492] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Bootstrap starting.
I20260621 02:53:17.416540 25090 tablet_bootstrap.cc:654] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:17.417591 25090 log.cc:824] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:17.418136 25091 tablet_bootstrap.cc:654] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:17.419435 25091 log.cc:824] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:17.419977 25093 tablet_bootstrap.cc:492] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Bootstrap starting.
I20260621 02:53:17.420689 25090 tablet_bootstrap.cc:492] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: No bootstrap required, opened a new log
I20260621 02:53:17.420957 25090 ts_tablet_manager.cc:1397] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:53:17.423213 25091 tablet_bootstrap.cc:492] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: No bootstrap required, opened a new log
I20260621 02:53:17.423408 25093 tablet_bootstrap.cc:654] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:17.423492 25091 ts_tablet_manager.cc:1397] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260621 02:53:17.424526 25093 log.cc:824] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:17.427368 25093 tablet_bootstrap.cc:492] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: No bootstrap required, opened a new log
I20260621 02:53:17.427614 25093 ts_tablet_manager.cc:1397] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260621 02:53:17.431074 25091 raft_consensus.cc:348] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.430985 25090 raft_consensus.cc:348] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.431396 25090 raft_consensus.cc:374] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:17.431408 25091 raft_consensus.cc:374] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:17.431502 25090 raft_consensus.cc:729] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0305e76492864f4ba9c0d55b1643114c, State: Initialized, Role: FOLLOWER
I20260621 02:53:17.431514 25091 raft_consensus.cc:729] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532f8c22bcd047aeb27f950316f6d92a, State: Initialized, Role: FOLLOWER
I20260621 02:53:17.432116 25091 consensus_queue.cc:260] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.432123 25090 consensus_queue.cc:260] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.433863 25091 ts_tablet_manager.cc:1428] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260621 02:53:17.434376 25090 ts_tablet_manager.cc:1428] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260621 02:53:17.434556 24783 heartbeater.cc:503] Master 127.11.109.62:46681 was elected leader, sending a full tablet report...
I20260621 02:53:17.439513 25093 raft_consensus.cc:348] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.439926 25093 raft_consensus.cc:374] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:17.440029 25093 raft_consensus.cc:729] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c102b00a2a4b4fb556b9950b55dad6, State: Initialized, Role: FOLLOWER
I20260621 02:53:17.440477 25093 consensus_queue.cc:260] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.441362 25062 heartbeater.cc:503] Master 127.11.109.62:46681 was elected leader, sending a full tablet report...
I20260621 02:53:17.441767 25093 ts_tablet_manager.cc:1428] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260621 02:53:17.462631 25097 raft_consensus.cc:482] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:17.462894 25097 raft_consensus.cc:504] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.464504 25097 leader_election.cc:288] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c102b00a2a4b4fb556b9950b55dad6 (127.11.109.3:35167), 532f8c22bcd047aeb27f950316f6d92a (127.11.109.2:39305)
I20260621 02:53:17.471457 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.464850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:55159 (local address 127.11.109.3:35167)
0621 02:53:17.465045 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.465051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.465916 (+ 865us) server_negotiation.cc:407] Connection header received
0621 02:53:17.466101 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.466105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.466159 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.466273 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:17.466964 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.467664 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.469715 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.470454 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.470600 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.470685 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.471077 (+ 392us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.471196 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:53:17.471495 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.465129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:36747 (local address 127.11.109.2:39305)
0621 02:53:17.465367 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.465374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.465849 (+ 475us) server_negotiation.cc:407] Connection header received
0621 02:53:17.466079 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.466083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.466140 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.466236 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:17.466935 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.467660 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.469742 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.470479 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.470659 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.470725 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.471042 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.471180 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 02:53:17.471688 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.464793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:39305 (local address 127.11.109.1:36747)
0621 02:53:17.465719 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:17.465742 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:17.465920 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:17.466298 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:17.466302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:17.466324 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:17.466786 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.466796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.467810 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.467815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.469575 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.469586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.470739 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.470744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.470891 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.470914 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:53:17.470987 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 02:53:17.471132 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":42,"thread_start_us":507,"threads_started":1}
I20260621 02:53:17.471796 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.464869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35167 (local address 127.11.109.1:55159)
0621 02:53:17.465831 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:17.465847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:17.465951 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:17.466286 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:17.466290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:17.466314 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:17.466772 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.466784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.467796 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.467804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.469561 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:17.469572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.470679 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.470686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:17.470854 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.470922 (+ 68us) client_negotiation.cc:769] Sending connection context
0621 02:53:17.471031 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 02:53:17.471160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":137,"thread_start_us":678,"threads_started":1}
I20260621 02:53:17.472666 24882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3633e9d6fe6a4a21b4ee9b59e56e3d60" candidate_uuid: "0305e76492864f4ba9c0d55b1643114c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532f8c22bcd047aeb27f950316f6d92a" is_pre_election: true
I20260621 02:53:17.472710 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3633e9d6fe6a4a21b4ee9b59e56e3d60" candidate_uuid: "0305e76492864f4ba9c0d55b1643114c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c102b00a2a4b4fb556b9950b55dad6" is_pre_election: true
I20260621 02:53:17.473124 24882 raft_consensus.cc:2454] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0305e76492864f4ba9c0d55b1643114c in term 0.
I20260621 02:53:17.473150 25017 raft_consensus.cc:2454] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0305e76492864f4ba9c0d55b1643114c in term 0.
I20260621 02:53:17.473805 24672 leader_election.cc:302] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0305e76492864f4ba9c0d55b1643114c, 71c102b00a2a4b4fb556b9950b55dad6; no voters:
I20260621 02:53:17.474201 25097 raft_consensus.cc:2781] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:53:17.474351 25097 raft_consensus.cc:482] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:53:17.474459 25097 raft_consensus.cc:3037] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:17.476773 25097 raft_consensus.cc:504] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.477393 25097 leader_election.cc:288] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [CANDIDATE]: Term 1 election: Requested vote from peers 71c102b00a2a4b4fb556b9950b55dad6 (127.11.109.3:35167), 532f8c22bcd047aeb27f950316f6d92a (127.11.109.2:39305)
I20260621 02:53:17.477727 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3633e9d6fe6a4a21b4ee9b59e56e3d60" candidate_uuid: "0305e76492864f4ba9c0d55b1643114c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c102b00a2a4b4fb556b9950b55dad6"
I20260621 02:53:17.477916 25017 raft_consensus.cc:3037] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:17.480137 25017 raft_consensus.cc:2454] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0305e76492864f4ba9c0d55b1643114c in term 1.
I20260621 02:53:17.480479 24672 leader_election.cc:302] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0305e76492864f4ba9c0d55b1643114c, 71c102b00a2a4b4fb556b9950b55dad6; no voters:
I20260621 02:53:17.480494 24882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3633e9d6fe6a4a21b4ee9b59e56e3d60" candidate_uuid: "0305e76492864f4ba9c0d55b1643114c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532f8c22bcd047aeb27f950316f6d92a"
I20260621 02:53:17.480731 24882 raft_consensus.cc:3037] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:17.480760 25097 raft_consensus.cc:2781] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:17.481258 25097 raft_consensus.cc:686] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 1 LEADER]: Becoming Leader. State: Replica: 0305e76492864f4ba9c0d55b1643114c, State: Running, Role: LEADER
I20260621 02:53:17.481710 25097 consensus_queue.cc:237] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } }
I20260621 02:53:17.483160 24882 raft_consensus.cc:2454] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0305e76492864f4ba9c0d55b1643114c in term 1.
I20260621 02:53:17.485608 24585 catalog_manager.cc:5495] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0305e76492864f4ba9c0d55b1643114c (127.11.109.1). New cstate: current_term: 1 leader_uuid: "0305e76492864f4ba9c0d55b1643114c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0305e76492864f4ba9c0d55b1643114c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 } health_report { overall_health: UNKNOWN } } }
W20260621 02:53:17.488849 24928 tablet.cc:2369] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:17.513814 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.508869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.11.109.1:34387)
0621 02:53:17.509052 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.509057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.509114 (+ 57us) server_negotiation.cc:407] Connection header received
0621 02:53:17.509353 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.509357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.509438 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.509554 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:53:17.509888 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.510588 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.511883 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.512254 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.512341 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.512799 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:53:17.513338 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:53:17.513450 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.513521 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.513591 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:53:17.521873 24882 raft_consensus.cc:1261] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 1 FOLLOWER]: Refusing update from remote peer 0305e76492864f4ba9c0d55b1643114c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:17.521888 25017 raft_consensus.cc:1261] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 1 FOLLOWER]: Refusing update from remote peer 0305e76492864f4ba9c0d55b1643114c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:17.522909 25101 consensus_queue.cc:1035] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [LEADER]: Connected to new peer: Peer: permanent_uuid: "532f8c22bcd047aeb27f950316f6d92a" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:17.523283 25097 consensus_queue.cc:1035] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c102b00a2a4b4fb556b9950b55dad6" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:17.532634 25104 mvcc.cc:205] Tried to move back new op lower bound from 7299114588239065088 to 7299114588088762368. Current Snapshot: MvccSnapshot[applied={T|T < 7299114588239065088}]
I20260621 02:53:17.538376 25105 mvcc.cc:205] Tried to move back new op lower bound from 7299114588239065088 to 7299114588088762368. Current Snapshot: MvccSnapshot[applied={T|T < 7299114588239065088}]
I20260621 02:53:17.549021 24585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:17.549419 24585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:17.552819 24585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 471e2c119ca544a3821c9ff2c61ef165: Sending DeleteTablet for 3 replicas of tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60
I20260621 02:53:17.553998 24997 tablet_service.cc:1514] Processing DeleteTablet for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:17 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:45782
I20260621 02:53:17.554003 24717 tablet_service.cc:1514] Processing DeleteTablet for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:17 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:48028
I20260621 02:53:17.554867 25114 tablet_replica.cc:330] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: stopping tablet replica
I20260621 02:53:17.555027 24862 tablet_service.cc:1514] Processing DeleteTablet for tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:17 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51722
I20260621 02:53:17.555308 25114 raft_consensus.cc:2229] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:17.555744 25114 raft_consensus.cc:2258] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:17.557837 25114 ts_tablet_manager.cc:1905] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:17.556941 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24623
I20260621 02:53:17.560276 25113 tablet_replica.cc:330] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: stopping tablet replica
I20260621 02:53:17.565169 25114 ts_tablet_manager.cc:1918] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:17.565841 25113 raft_consensus.cc:2229] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:53:17.570019 25115 tablet_replica.cc:330] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: stopping tablet replica
I20260621 02:53:17.566644 25113 raft_consensus.cc:2258] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:17.570453 25115 raft_consensus.cc:2229] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:17.571048 25115 raft_consensus.cc:2258] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:17.571518 25114 log.cc:1196] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/3633e9d6fe6a4a21b4ee9b59e56e3d60
I20260621 02:53:17.572168 25114 ts_tablet_manager.cc:1939] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 71c102b00a2a4b4fb556b9950b55dad6: Deleting consensus metadata
I20260621 02:53:17.572525 25113 ts_tablet_manager.cc:1905] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:17.574162 24572 catalog_manager.cc:4841] TS 71c102b00a2a4b4fb556b9950b55dad6 (127.11.109.3:35167): tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 (table test-table [id=a311cd47f25947d7a2ea09ab4c1d12a4]) successfully deleted
I20260621 02:53:17.575626 25115 ts_tablet_manager.cc:1905] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:17.580607 25113 ts_tablet_manager.cc:1918] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:17.580825 25113 log.cc:1196] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/3633e9d6fe6a4a21b4ee9b59e56e3d60
I20260621 02:53:17.581439 25113 ts_tablet_manager.cc:1939] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 0305e76492864f4ba9c0d55b1643114c: Deleting consensus metadata
I20260621 02:53:17.583199 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.559580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.11.109.1:34387)
0621 02:53:17.559745 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.559751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.559766 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:53:17.559833 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.559841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.559918 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.560020 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:17.565925 (+ 5905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.567446 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.569510 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.569931 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.570072 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.573689 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:17.573701 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:17.573716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:17.573760 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:17.578757 (+ 4997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.579731 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.579739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.579743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.579879 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.580322 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.580330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.580333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.580845 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:17.581364 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.582568 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.582793 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:53:17.583207 24570 catalog_manager.cc:4841] TS 0305e76492864f4ba9c0d55b1643114c (127.11.109.1:34387): tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 (table test-table [id=a311cd47f25947d7a2ea09ab4c1d12a4]) successfully deleted
I20260621 02:53:17.584095 24778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:17.589224 25115 ts_tablet_manager.cc:1918] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:17.589422 25115 log.cc:1196] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/3633e9d6fe6a4a21b4ee9b59e56e3d60
I20260621 02:53:17.589849 25115 ts_tablet_manager.cc:1939] T 3633e9d6fe6a4a21b4ee9b59e56e3d60 P 532f8c22bcd047aeb27f950316f6d92a: Deleting consensus metadata
I20260621 02:53:17.591236 24571 catalog_manager.cc:4841] TS 532f8c22bcd047aeb27f950316f6d92a (127.11.109.2:39305): tablet 3633e9d6fe6a4a21b4ee9b59e56e3d60 (table test-table [id=a311cd47f25947d7a2ea09ab4c1d12a4]) successfully deleted
I20260621 02:53:17.680310 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24623
I20260621 02:53:17.694705 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24787
I20260621 02:53:17.708287 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.695430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.11.109.2:39305)
0621 02:53:17.695609 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.695615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.695712 (+ 97us) server_negotiation.cc:407] Connection header received
0621 02:53:17.695934 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.695948 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.696011 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.696114 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:17.697475 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.698193 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.699454 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.699859 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.699986 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.702570 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:17.702584 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:17.702599 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:17.702636 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:17.705524 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.706200 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.706207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.706210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.706341 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.706759 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.706766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.706769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.707107 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:17.707422 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.707721 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.707927 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:53:17.708928 24922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:17.800736 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24787
I20260621 02:53:17.817135 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24929
I20260621 02:53:17.829435 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:17.817914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.11.109.3:35167)
0621 02:53:17.818093 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:17.818097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:17.818115 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:53:17.818251 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:17.818255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:17.818312 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:17.818391 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:17.819576 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.820269 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.821479 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:17.821852 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:17.821939 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:17.824302 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:17.824316 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:17.824333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:17.824375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:17.827003 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.827596 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.827603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.827605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.827718 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:17.828001 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:17.828004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:17.828006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:17.828331 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:17.828566 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:17.828783 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 02:53:17.828930 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:53:17.830071 25057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:17.925040 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24929
I20260621 02:53:17.941097 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 24553
I20260621 02:53:17.942195 24616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:18.053613 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 24553
I20260621 02:53:18.078696 11700 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 02:53:18.414103 11700 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 02:53:18.414362 11700 mini_ranger.cc:66] Stopping Ranger...
I20260621 02:53:18.759250 11700 mini_ranger.cc:68] Stopped Ranger
2026-06-21 02:53:18.759 UTC [23605] LOG: received smart shutdown request
2026-06-21 02:53:18.762 UTC [23605] LOG: background worker "logical replication launcher" (PID 23614) exited with exit code 1
2026-06-21 02:53:18.762 UTC [23609] LOG: shutting down
2026-06-21 02:53:18.897 UTC [23605] LOG: database system is shut down
2026-06-21T02:53:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50680 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:53:19 dist-test-slave-bc52 krb5kdc[25128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:53:19 dist-test-slave-bc52 krb5kdc[25128](info): set up 2 sockets
Jun 21 02:53:19 dist-test-slave-bc52 krb5kdc[25128](info): commencing operation
krb5kdc: starting...
W20260621 02:53:20.109619 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:53:20 dist-test-slave-bc52 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:53:20.317499 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44267
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:44267,127.11.109.61:43961,127.11.109.60:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:20.642088 25141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:20.642444 25141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:20.642637 25141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:20.650295 25141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:20.650384 25141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:20.650429 25141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:20.650486 25141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:20.650532 25141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:20.660085 25141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:44267,127.11.109.61:43961,127.11.109.60:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:20.661350 25141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:20.662575 25141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:20.673833 25147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:20.673810 25149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:20.673934 25146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:20.674475 25141 server_base.cc:1034] running on GCE node
I20260621 02:53:20.675251 25141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:20.676560 25141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:20.677750 25141 hybrid_clock.cc:648] HybridClock initialized: now 1782010400677716 us; error 34 us; skew 500 ppm
I20260621 02:53:20.678036 25141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:20 dist-test-slave-bc52 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:20.680900 25141 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:20.682355 25141 webserver.cc:460] Webserver started at http://127.11.109.62:40253/ using document root <none> and password file <none>
I20260621 02:53:20.682957 25141 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:20.683055 25141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:20.683316 25141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:20.686121 25141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "14ef3da058e84bf0a2f96540fb47e143"
format_stamp: "Formatted at 2026-06-21 02:53:20 on dist-test-slave-bc52"
server_key: "a98b507cd03d2b429817ed919fb28061"
server_key_iv: "74569134d88fc273eb8e6020fa206efd"
server_key_version: "encryptionkey@0"
I20260621 02:53:20.686834 25141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "14ef3da058e84bf0a2f96540fb47e143"
format_stamp: "Formatted at 2026-06-21 02:53:20 on dist-test-slave-bc52"
server_key: "a98b507cd03d2b429817ed919fb28061"
server_key_iv: "74569134d88fc273eb8e6020fa206efd"
server_key_version: "encryptionkey@0"
I20260621 02:53:20.691605 25141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:53:20.694983 25156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:20.696326 25141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:53:20.696504 25141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "14ef3da058e84bf0a2f96540fb47e143"
format_stamp: "Formatted at 2026-06-21 02:53:20 on dist-test-slave-bc52"
server_key: "a98b507cd03d2b429817ed919fb28061"
server_key_iv: "74569134d88fc273eb8e6020fa206efd"
server_key_version: "encryptionkey@0"
I20260621 02:53:20.696691 25141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:20.711175 25141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:20.715078 25141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:20.715364 25141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:20.734112 25141 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:44267
I20260621 02:53:20.734143 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:44267 every 8 connection(s)
I20260621 02:53:20.735553 25141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:20.739079 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:20.745862 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25141
I20260621 02:53:20.745339 25208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:53:20.746151 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:53:20.746768 11700 external_mini_cluster.cc:1428] Setting key 83a17a56fa170168b23dc7bbb598aa4b
Jun 21 02:53:20 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.61@KRBTEST.COM; defaulting to no policy
I20260621 02:53:20.769377 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.748447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.11.109.62:44267)
0621 02:53:20.750155 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:20.750163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:20.750214 (+ 51us) server_negotiation.cc:407] Connection header received
0621 02:53:20.751679 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:20.751684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:20.751902 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:20.752107 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:20.753400 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:20.754833 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:20.756196 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:20.757209 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:20.757411 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:20.760838 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:20.760849 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:20.760866 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:20.760907 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:20.764917 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:20.765868 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:20.765878 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:20.765882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:20.766078 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:20.766485 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:20.766492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:20.766497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:20.767236 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:20.767444 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:20.768017 (+ 573us) server_negotiation.cc:299] Negotiation successful
0621 02:53:20.768340 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":1502,"spinlock_wait_cycles":14720,"thread_start_us":1286,"threads_started":1}
I20260621 02:53:20.769466 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.748446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44267 (local address 127.0.0.1:50970)
0621 02:53:20.749318 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.749370 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:20.751324 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:20.752124 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:20.752130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:20.752818 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:20.753133 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:20.753150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:20.755000 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:20.755004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:20.756053 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:20.756064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:20.757425 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:20.757430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:20.757543 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:20.758239 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:20.758273 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:20.760601 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:20.765068 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:20.765077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:20.765086 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:20.765733 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:20.766212 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:20.766216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:20.766218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:20.766358 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:20.767379 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:20.767385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:20.767566 (+ 181us) client_negotiation.cc:769] Sending connection context
0621 02:53:20.767830 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:53:20.768214 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.61@KRBTEST.COM" created.
I20260621 02:53:20.772524 25208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:53:20.773783 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.773281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47144)
0621 02:53:20.773486 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.773622 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 02:53:20.774082 25157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
I20260621 02:53:20.775743 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.775283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47154)
0621 02:53:20.775467 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.775560 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 02:53:20.775954 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:43961: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab.
Entry for principal kudu/127.11.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab.
I20260621 02:53:20.794588 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:43961
--webserver_interface=127.11.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab
--principal=kudu/127.11.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:44267,127.11.109.61:43961,127.11.109.60:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:20.825325 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } attempt: 1
I20260621 02:53:20.826431 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.825983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47166)
0621 02:53:20.826181 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.826297 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:53:20.827443 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.827083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47182)
0621 02:53:20.827229 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.827297 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:53:20.827630 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:43961: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
I20260621 02:53:20.895879 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } attempt: 2
I20260621 02:53:20.897156 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.896710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47196)
0621 02:53:20.896924 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.897015 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260621 02:53:20.898211 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.897875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47206)
0621 02:53:20.898021 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.898092 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:53:20.898402 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:43961: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
I20260621 02:53:20.989660 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } attempt: 3
I20260621 02:53:20.990875 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.990459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47220)
0621 02:53:20.990652 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.990742 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:53:20.991918 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:20.991570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47232)
0621 02:53:20.991716 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:20.991794 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:53:20.992094 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:43961: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
W20260621 02:53:21.114049 25215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:21.114356 25215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:21.114518 25215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:21.121491 25215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:21.121651 25215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:21.121754 25215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:21.121860 25215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:21.121932 25215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:21.131117 25215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:44267,127.11.109.61:43961,127.11.109.60:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab
--principal=kudu/127.11.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:21.132344 25215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:21.133502 25215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:53:21.135299 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } attempt: 4
I20260621 02:53:21.136451 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.135952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47248)
0621 02:53:21.136223 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.136311 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260621 02:53:21.137521 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.137095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47254)
0621 02:53:21.137320 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.137405 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:53:21.137699 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:43961: Network error: Client connection negotiation failed: client connection to 127.11.109.61:43961: connect: Connection refused (error 111)
W20260621 02:53:21.145052 25220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:21.145098 25223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:21.145140 25221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:21.150470 25215 server_base.cc:1034] running on GCE node
I20260621 02:53:21.151162 25215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:21.152451 25215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:21.153687 25215 hybrid_clock.cc:648] HybridClock initialized: now 1782010401153585 us; error 70 us; skew 500 ppm
I20260621 02:53:21.154038 25215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:21.157002 25215 init.cc:377] Logged in from keytab as kudu/127.11.109.61@KRBTEST.COM (short username kudu)
I20260621 02:53:21.158478 25215 webserver.cc:460] Webserver started at http://127.11.109.61:38019/ using document root <none> and password file <none>
I20260621 02:53:21.159170 25215 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:21.159312 25215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:21.159601 25215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:21.162529 25215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data/instance:
uuid: "7ead181889ad4168bfa62345beae5cc3"
format_stamp: "Formatted at 2026-06-21 02:53:21 on dist-test-slave-bc52"
server_key: "a1c2a42264c2f49362dc8266777144d9"
server_key_iv: "f9104b7679ed863ffb7794f6ab54c6bc"
server_key_version: "encryptionkey@0"
I20260621 02:53:21.163329 25215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal/instance:
uuid: "7ead181889ad4168bfa62345beae5cc3"
format_stamp: "Formatted at 2026-06-21 02:53:21 on dist-test-slave-bc52"
server_key: "a1c2a42264c2f49362dc8266777144d9"
server_key_iv: "f9104b7679ed863ffb7794f6ab54c6bc"
server_key_version: "encryptionkey@0"
I20260621 02:53:21.168154 25215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 02:53:21.171391 25230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:21.172587 25215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:53:21.172768 25215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal
uuid: "7ead181889ad4168bfa62345beae5cc3"
format_stamp: "Formatted at 2026-06-21 02:53:21 on dist-test-slave-bc52"
server_key: "a1c2a42264c2f49362dc8266777144d9"
server_key_iv: "f9104b7679ed863ffb7794f6ab54c6bc"
server_key_version: "encryptionkey@0"
I20260621 02:53:21.172925 25215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:21.194631 25215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:21.198278 25215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:21.198495 25215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:21.216862 25215 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.61:43961
I20260621 02:53:21.216936 25281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.61:43961 every 8 connection(s)
I20260621 02:53:21.218178 25215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
I20260621 02:53:21.221700 25282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:21.223613 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25215
I20260621 02:53:21.223800 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-1/wal/instance
I20260621 02:53:21.224224 11700 external_mini_cluster.cc:1428] Setting key 8be88e084ee8deb948f6a84c5d5b6ef3
I20260621 02:53:21.227768 25282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.60@KRBTEST.COM" created.
I20260621 02:53:21.250288 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.230153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.11.109.62:44267)
0621 02:53:21.230398 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.230404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.231540 (+ 1136us) server_negotiation.cc:407] Connection header received
0621 02:53:21.234354 (+ 2814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.234358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.234420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.234501 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.236443 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.237098 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.239601 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.240023 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.240120 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.244228 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.244236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.244244 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.244287 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.247207 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.247945 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.247952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.247960 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.248074 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.248486 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.248490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.248493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.248810 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.248962 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.249728 (+ 766us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.249966 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:53:21.251228 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.230593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44267 (local address 127.0.0.1:50984)
0621 02:53:21.231366 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.231410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.234059 (+ 2649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.234742 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.234755 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.235473 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.236188 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.236226 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.237289 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.237297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.239380 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.239396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.240265 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.240276 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.240601 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.241320 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.241354 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.243990 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.247340 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.247347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.247359 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.247819 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.248186 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.248191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.248194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.248368 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.248947 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.248955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.249371 (+ 416us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.249622 (+ 251us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.250084 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":301,"threads_started":1}
I20260621 02:53:21.253204 25282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:53:21.270062 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.254027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47268)
0621 02:53:21.254227 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.254267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.254383 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.255381 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.255388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.255891 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.256095 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.256109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.257504 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.257512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.259059 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.259076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.259858 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.259866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.260103 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.260768 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.260784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.263012 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.267446 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.267450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.267453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.267838 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.268252 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.268255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.268257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.268409 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.269087 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.269092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.269293 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.269407 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.269680 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 02:53:21.270110 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.254290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.11.109.61:43961)
0621 02:53:21.254967 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.254972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.254997 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:53:21.255039 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.255045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.255211 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.255326 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.256249 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.257369 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.259203 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.259697 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.259802 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.263249 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.263265 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.263293 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.263342 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.267317 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.267976 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.267988 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.267994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.268140 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.268499 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.268502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.268504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.268885 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.269132 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.269588 (+ 456us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.269767 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":387,"threads_started":1}
I20260621 02:53:21.272825 25282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:53:21.274037 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.273588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53902)
0621 02:53:21.273775 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.273901 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:53:21.274328 25232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab.
Entry for principal kudu/127.11.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab.
I20260621 02:53:21.275996 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.275579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53916)
0621 02:53:21.275740 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.275863 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:53:21.276259 25282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.276588 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:33591
--webserver_interface=127.11.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab
--principal=kudu/127.11.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:44267,127.11.109.61:43961,127.11.109.60:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:21.325593 25282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 1
I20260621 02:53:21.326884 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.326395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53918)
0621 02:53:21.326616 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.326730 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 02:53:21.327992 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.327526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53932)
0621 02:53:21.327743 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.327845 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260621 02:53:21.328205 25282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.396458 25282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 2
I20260621 02:53:21.397701 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.397254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53934)
0621 02:53:21.397458 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.397547 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:53:21.398725 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.398434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53936)
0621 02:53:21.398553 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.398620 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 02:53:21.399024 25282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.437040 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } attempt: 5
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:53:21.451601 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.437982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.11.109.61:43961)
0621 02:53:21.438346 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.438350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.438365 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:53:21.438412 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.438416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.438486 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.438560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.439785 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.440469 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.442536 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.442954 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.443046 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.446487 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.446492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.446495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.446525 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.449025 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.449663 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.449670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.449673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.449794 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.450230 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.450233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.450235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.450501 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.450656 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.451034 (+ 378us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.451240 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260621 02:53:21.451627 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.437847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:47284)
0621 02:53:21.438041 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.438064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.438168 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.438665 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.438679 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.439248 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.439568 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.439585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.440623 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.440630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.442375 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.442394 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.443111 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.443118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.443332 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.444000 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.444023 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.446296 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.449161 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.449167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.449188 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.449542 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.449905 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.449910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.449914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.450047 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.450620 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.450625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.450789 (+ 164us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.450927 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.451223 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:53:21.453014 25208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:53:21.454064 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.453700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53938)
0621 02:53:21.453860 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.453942 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 02:53:21.455006 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.454704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:53948)
0621 02:53:21.454842 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.454904 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:53:21.455267 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.490259 25282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 3
I20260621 02:53:21.491312 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.490872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56410)
0621 02:53:21.491079 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.491161 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 02:53:21.492331 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.491949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56418)
0621 02:53:21.492160 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.492225 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260621 02:53:21.492549 25282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.506500 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 1
I20260621 02:53:21.507479 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.507068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56420)
0621 02:53:21.507259 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.507338 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 02:53:21.508399 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.508073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56436)
0621 02:53:21.508226 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.508303 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:53:21.508605 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.576867 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 2
I20260621 02:53:21.578055 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.577515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56438)
0621 02:53:21.577749 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.577875 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:53:21.579062 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.578767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56452)
0621 02:53:21.578896 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.578960 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:53:21.579290 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
W20260621 02:53:21.597761 25289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:21.598038 25289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:21.598201 25289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:21.605611 25289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:21.605727 25289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:21.605813 25289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:21.605916 25289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:21.605994 25289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:21.615231 25289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:44267,127.11.109.61:43961,127.11.109.60:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab
--principal=kudu/127.11.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:33591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:21.616509 25289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:21.617813 25289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:21.629209 25294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:21.629300 25295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:21.629334 25297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:21.630069 25289 server_base.cc:1034] running on GCE node
I20260621 02:53:21.630815 25289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:21.632238 25289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:21.633457 25289 hybrid_clock.cc:648] HybridClock initialized: now 1782010401633422 us; error 49 us; skew 500 ppm
I20260621 02:53:21.633770 25289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:53:21.635771 25282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 4
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:21.636919 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.636327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56454)
0621 02:53:21.636584 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.636688 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 02:53:21.637039 25289 init.cc:377] Logged in from keytab as kudu/127.11.109.60@KRBTEST.COM (short username kudu)
I20260621 02:53:21.638127 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.637746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56462)
0621 02:53:21.637908 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.637993 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 02:53:21.638370 25282 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.638846 25289 webserver.cc:460] Webserver started at http://127.11.109.60:37973/ using document root <none> and password file <none>
I20260621 02:53:21.639525 25289 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:21.639648 25289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:21.639942 25289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:21.643031 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data/instance:
uuid: "e7da00a7d25d4cdb8d1dee6755c32b72"
format_stamp: "Formatted at 2026-06-21 02:53:21 on dist-test-slave-bc52"
server_key: "53ebaddbfdbc7ac65ec89c2842c014c4"
server_key_iv: "d3134c3a6d0b98c5ee650db82452c999"
server_key_version: "encryptionkey@0"
I20260621 02:53:21.643868 25289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal/instance:
uuid: "e7da00a7d25d4cdb8d1dee6755c32b72"
format_stamp: "Formatted at 2026-06-21 02:53:21 on dist-test-slave-bc52"
server_key: "53ebaddbfdbc7ac65ec89c2842c014c4"
server_key_iv: "d3134c3a6d0b98c5ee650db82452c999"
server_key_version: "encryptionkey@0"
I20260621 02:53:21.648653 25289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:53:21.651818 25304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:21.653028 25289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:53:21.653249 25289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal
uuid: "e7da00a7d25d4cdb8d1dee6755c32b72"
format_stamp: "Formatted at 2026-06-21 02:53:21 on dist-test-slave-bc52"
server_key: "53ebaddbfdbc7ac65ec89c2842c014c4"
server_key_iv: "d3134c3a6d0b98c5ee650db82452c999"
server_key_version: "encryptionkey@0"
I20260621 02:53:21.653450 25289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:21.667975 25289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:21.672235 25289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:21.672513 25289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:21.685510 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 3
I20260621 02:53:21.686635 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.686224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56468)
0621 02:53:21.686413 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.686494 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:53:21.687804 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.687369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56484)
0621 02:53:21.687601 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.687695 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260621 02:53:21.688053 25208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:33591: Network error: Client connection negotiation failed: client connection to 127.11.109.60:33591: connect: Connection refused (error 111)
I20260621 02:53:21.690678 25289 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.60:33591
I20260621 02:53:21.690716 25355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.60:33591 every 8 connection(s)
I20260621 02:53:21.691911 25289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
I20260621 02:53:21.695477 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:21.696244 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25289
I20260621 02:53:21.696440 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782010204918124-11700-0/minicluster-data/master-2/wal/instance
I20260621 02:53:21.696928 11700 external_mini_cluster.cc:1428] Setting key 79c187f1d79650ec74e2b60268ea3eee
I20260621 02:53:21.704828 25356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:21.720538 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.698725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.11.109.60:33591)
0621 02:53:21.699517 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.699533 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.699957 (+ 424us) server_negotiation.cc:407] Connection header received
0621 02:53:21.702560 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.702582 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.702967 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.703377 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.704826 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.706206 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.707688 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.708511 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.708676 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.713441 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.713450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.713463 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.713538 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.716699 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.717401 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.717411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.717420 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.717602 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.717989 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.717996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.718000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.718466 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.718775 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.719199 (+ 424us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.719574 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":354,"threads_started":1}
I20260621 02:53:21.722784 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.707161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.11.109.62:44267)
0621 02:53:21.707376 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.707383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.708159 (+ 776us) server_negotiation.cc:407] Connection header received
0621 02:53:21.708320 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.708325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.708420 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.708522 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.709929 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.710611 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.712315 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.713024 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.713225 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.716921 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.716926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.716928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.716958 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.719878 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.720585 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.720592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.720596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.720728 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.721245 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.721250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.721253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.721604 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.721724 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.722220 (+ 496us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.722424 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 02:53:21.722831 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.707173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44267 (local address 127.0.0.1:48594)
0621 02:53:21.708005 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.708047 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.708196 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.708609 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.708615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.709393 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.709773 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.709786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.710775 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.710781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.712131 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.712142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.713264 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.713269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.713469 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.714138 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.714168 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.716702 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.719999 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.720012 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.720019 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.720458 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.720902 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.720906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.720909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.721090 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.721729 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.721736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.721919 (+ 183us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.722111 (+ 192us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.722415 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":454,"threads_started":1}
I20260621 02:53:21.724371 25356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:21.738750 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.725099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:43961 (local address 127.0.0.1:57438)
0621 02:53:21.725263 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.725279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.725375 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.725750 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.725754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.726258 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.726448 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.726459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.727629 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.727633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.728681 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.728696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.729536 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.729540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.729795 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.730342 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.730357 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.732476 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.736132 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.736138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.736142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.736581 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.737004 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.737011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.737016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.737161 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.737822 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.737826 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.737999 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.738151 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.738410 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:53:21.738830 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.725123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.11.109.61:43961)
0621 02:53:21.725291 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.725295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.725344 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:53:21.725499 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.725504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.725587 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.725692 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.726619 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.727489 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.728825 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.729405 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.729540 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.732656 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.732662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.732666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.732711 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.736000 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.736703 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.736709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.736712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.736892 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.737353 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.737358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.737362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.737677 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.737833 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.738242 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.738444 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 02:53:21.739998 25356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:53:21.740285 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.723789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.11.109.62:44267)
0621 02:53:21.723921 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.723925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.723938 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:53:21.723972 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.723975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.724021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.724092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.725767 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.726928 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.728341 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.728962 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.729047 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.734203 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.734211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.734215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.734244 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.737395 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.738109 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.738114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.738117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.738264 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.738661 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.738665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.738669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.739013 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.739464 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.739751 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.739938 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:53:21.753780 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.740776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56510)
0621 02:53:21.740930 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.740948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.741039 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.741665 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.741670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.742088 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.742296 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.742308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.743637 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.743642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.744621 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.744633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.745354 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.745358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.745519 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.746060 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.746075 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.748546 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.751498 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.751502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.751504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.751942 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.752339 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.752342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.752345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.752459 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.753019 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.753023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.753151 (+ 128us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.753345 (+ 194us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.753541 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:53:21.754066 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.741272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.11.109.60:33591)
0621 02:53:21.741420 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.741427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.741445 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:53:21.741484 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.741489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.741545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.741632 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.742462 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.743502 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.744750 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.745220 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.745359 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.748723 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.748727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.748730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.748755 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.751381 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.752057 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.752062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.752066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.752204 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.752556 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.752561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.752565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.752898 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.753027 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.753469 (+ 442us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.753707 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260621 02:53:21.757560 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.742052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.11.109.61:43961)
0621 02:53:21.742206 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.742214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.742576 (+ 362us) server_negotiation.cc:407] Connection header received
0621 02:53:21.742616 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.742619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.742667 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.742766 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.743989 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.744980 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.746313 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.746923 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.747051 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.751202 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.751209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.751213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.751248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.754439 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.755611 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.755614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.755616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.755732 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.756129 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.756132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.756133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.756410 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.756608 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.756912 (+ 304us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.757099 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:53:21.759763 25356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: Bootstrap starting.
I20260621 02:53:21.762084 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:53:21.763401 25356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:21.764372 25356 log.cc:824] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:21.767269 25356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: No bootstrap required, opened a new log
I20260621 02:53:21.788515 25356 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.794724 25356 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:21.794941 25356 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7da00a7d25d4cdb8d1dee6755c32b72, State: Initialized, Role: FOLLOWER
I20260621 02:53:21.795892 25356 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.801563 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.767193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.11.109.60:33591)
0621 02:53:21.767382 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.767388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.778008 (+ 10620us) server_negotiation.cc:407] Connection header received
0621 02:53:21.778689 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.778696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.778785 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.778887 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.782902 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.784381 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.787715 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.788467 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.788600 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.793369 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.793377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.793382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.793418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.797618 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.798788 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.798795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.798798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.798942 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.799412 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.799420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.799424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.799843 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.800278 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.800853 (+ 575us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.801076 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:53:21.801826 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.767050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.11.109.62:44267)
0621 02:53:21.767382 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.767389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.782775 (+ 15386us) server_negotiation.cc:407] Connection header received
0621 02:53:21.782943 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.782951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.783096 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.783272 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.785885 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.786990 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.789259 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.790075 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.790177 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.794617 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.794624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.794632 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.794669 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.798357 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.799231 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.799240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.799244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.799406 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.800000 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.800016 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.800031 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.800401 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.800829 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.801078 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.801462 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260621 02:53:21.805788 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:21.806550 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:21.807243 25356 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:21.811956 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.767220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.11.109.61:43961)
0621 02:53:21.767552 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.767561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.792094 (+ 24533us) server_negotiation.cc:407] Connection header received
0621 02:53:21.792400 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.792407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.792498 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.792573 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.794667 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.796071 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.798316 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.799116 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.799256 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.804090 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.804098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.804105 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.804153 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.807787 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.808788 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.808796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.808801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.808962 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.809459 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.809482 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.809487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.809911 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.810196 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.811345 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.811584 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
W20260621 02:53:21.826462 25380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:53:21.826596 25380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:53:21.822541 25289 master_runner.cc:186] Error getting master registration for 127.11.109.62:44267: OK, instance_id { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" instance_seqno: 1782010400728543 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:53:21.831658 25289 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:53:21.861977 25369 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:21.862264 25369 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.863682 25369 leader_election.cc:288] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ef3da058e84bf0a2f96540fb47e143 (127.11.109.62:44267), 7ead181889ad4168bfa62345beae5cc3 (127.11.109.61:43961)
I20260621 02:53:21.865264 25208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 4
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010400, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:53:21.883297 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.865772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56542)
0621 02:53:21.865955 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.865978 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.866088 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.866495 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.866499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.867082 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.867287 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.867298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.868840 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.868846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.870454 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.870469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.871460 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.871466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.871692 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.872335 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.872354 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.875229 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.880471 (+ 5242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.880476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.880498 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.881023 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.881453 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.881473 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.881477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.881655 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.882247 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.882253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.882431 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.882587 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.882881 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:53:21.883327 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.865826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.11.109.60:33591)
0621 02:53:21.865996 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.866000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.866022 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:53:21.866253 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.866258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.866356 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.866453 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.867433 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.868662 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.870612 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.871319 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.871441 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.875419 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.875426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.875431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.875474 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.880338 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.881133 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.881139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.881142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.881335 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.881770 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.881774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.881778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.882118 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.882263 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.882685 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.882927 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:53:21.887854 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: Bootstrap starting.
I20260621 02:53:21.891716 25208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:21.893029 25208 log.cc:824] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:21.902530 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: No bootstrap required, opened a new log
I20260621 02:53:21.914400 25208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.914790 25208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:21.914903 25208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ef3da058e84bf0a2f96540fb47e143, State: Initialized, Role: FOLLOWER
I20260621 02:53:21.915532 25208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.916558 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:21.916953 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:21.917758 25208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:21.917863 25183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ef3da058e84bf0a2f96540fb47e143" is_pre_election: true
I20260621 02:53:21.918470 25183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7da00a7d25d4cdb8d1dee6755c32b72 in term 0.
I20260621 02:53:21.919353 25307 leader_election.cc:302] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ef3da058e84bf0a2f96540fb47e143, e7da00a7d25d4cdb8d1dee6755c32b72; no voters:
I20260621 02:53:21.919812 25369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:53:21.920011 25369 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:53:21.920143 25369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 0 FOLLOWER]: Advancing to term 1
W20260621 02:53:21.924463 25310 tablet.cc:2369] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:21.924619 25369 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.928642 25183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ef3da058e84bf0a2f96540fb47e143"
I20260621 02:53:21.928925 25183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:21.933215 25183 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7da00a7d25d4cdb8d1dee6755c32b72 in term 1.
I20260621 02:53:21.933713 25307 leader_election.cc:302] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ef3da058e84bf0a2f96540fb47e143, e7da00a7d25d4cdb8d1dee6755c32b72; no voters:
I20260621 02:53:21.933722 25369 leader_election.cc:288] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [CANDIDATE]: Term 1 election: Requested vote from peers 14ef3da058e84bf0a2f96540fb47e143 (127.11.109.62:44267), 7ead181889ad4168bfa62345beae5cc3 (127.11.109.61:43961)
I20260621 02:53:21.934051 25369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:21.934213 25369 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [term 1 LEADER]: Becoming Leader. State: Replica: e7da00a7d25d4cdb8d1dee6755c32b72, State: Running, Role: LEADER
I20260621 02:53:21.934890 25369 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.937593 25282 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } attempt: 5
I20260621 02:53:21.946070 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7da00a7d25d4cdb8d1dee6755c32b72. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:21.946493 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:21.950234 25395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 21 02:53:21 dist-test-slave-bc52 krb5kdc[25128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010401, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:53:21.952409 25395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260621 02:53:21.967962 25162 tablet.cc:2369] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:21.968321 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.938268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:33591 (local address 127.0.0.1:56548)
0621 02:53:21.938685 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:21.938708 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:21.938840 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:21.939369 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:21.939377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:21.939900 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:21.940103 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.940115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.941618 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.941624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.943099 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:21.943114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.947377 (+ 4263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.947386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:21.947618 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.948376 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:21.948399 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:21.951552 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:21.957751 (+ 6199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.957757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.957762 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.958622 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.959857 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:21.959883 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:21.959888 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:21.960111 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:21.962261 (+ 2150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:21.962265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:21.962547 (+ 282us) client_negotiation.cc:769] Sending connection context
0621 02:53:21.962771 (+ 224us) client_negotiation.cc:240] Negotiation successful
0621 02:53:21.963277 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260621 02:53:21.971109 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:21.938608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.11.109.60:33591)
0621 02:53:21.938768 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:21.938774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:21.938802 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:53:21.938961 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:21.938967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:21.939049 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:21.939147 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:21.940272 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.941407 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.945878 (+ 4471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:21.946633 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:21.946799 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:21.951819 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:21.951826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:21.951831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:21.951868 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:21.957419 (+ 5551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.958988 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.958999 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.959003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.959527 (+ 524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:21.960437 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:21.960442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:21.960446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:21.961914 (+ 1468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:21.967008 (+ 5094us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:21.967633 (+ 625us) server_negotiation.cc:299] Negotiation successful
0621 02:53:21.969555 (+ 1922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:53:21.977279 25282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: Bootstrap starting.
I20260621 02:53:21.985201 25282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: Neither blocks nor log segments found. Creating new log.
W20260621 02:53:21.989770 25307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.109.62:44267, user_credentials={real_user=slave}} blocked reactor thread for 56065.7us
I20260621 02:53:21.991169 25282 log.cc:824] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:21.991907 25183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [term 1 FOLLOWER]: Refusing update from remote peer e7da00a7d25d4cdb8d1dee6755c32b72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:21.993049 25391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:21.996124 25282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: No bootstrap required, opened a new log
I20260621 02:53:22.010051 25282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:22.010514 25282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:22.010641 25282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ead181889ad4168bfa62345beae5cc3, State: Initialized, Role: FOLLOWER
I20260621 02:53:22.011456 25282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } }
I20260621 02:53:21.998378 25141 master_runner.cc:186] Error getting master registration for 127.11.109.61:43961: OK, instance_id { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" instance_seqno: 1782010401211486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:53:22.014062 25256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ead181889ad4168bfa62345beae5cc3"
I20260621 02:53:22.014365 25255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:22.014870 25257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ead181889ad4168bfa62345beae5cc3" is_pre_election: true
I20260621 02:53:22.012959 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.015516 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:22.015671 25282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:22.016973 25141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:53:22.018646 25255 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [term 1 FOLLOWER]: Refusing update from remote peer e7da00a7d25d4cdb8d1dee6755c32b72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:22.019590 25391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:53:22.027777 25397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:53:22.027915 25397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:53:22.033643 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7da00a7d25d4cdb8d1dee6755c32b72. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.034003 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:22.036818 25398 mvcc.cc:205] Tried to move back new op lower bound from 7299114606507651072 to 7299114606330847232. Current Snapshot: MvccSnapshot[applied={T|T < 7299114606507651072}]
I20260621 02:53:22.050148 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.050383 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:22.050662 25395 catalog_manager.cc:1347] Generated new cluster ID: 4f988c5b3fc04683b3b551f988c83e09
I20260621 02:53:22.050778 25395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:22.050740 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.051023 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:22.077016 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7da00a7d25d4cdb8d1dee6755c32b72. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.077387 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:22.079157 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.079505 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72 [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:22.085783 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14ef3da058e84bf0a2f96540fb47e143" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44267 } } peers { permanent_uuid: "7ead181889ad4168bfa62345beae5cc3" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 43961 } } peers { permanent_uuid: "e7da00a7d25d4cdb8d1dee6755c32b72" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 33591 } } }
I20260621 02:53:22.086004 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:53:22.102056 25395 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:53:22.106011 25395 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:22.124253 25419 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:53:22.130858 25395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7da00a7d25d4cdb8d1dee6755c32b72: Generated new TSK 0
I20260621 02:53:22.131942 25395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:53:23.030088 25397 catalog_manager.cc:1259] Loaded cluster ID: 4f988c5b3fc04683b3b551f988c83e09
I20260621 02:53:23.030194 25397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: loading cluster ID for follower catalog manager: success
I20260621 02:53:23.032618 25397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: acquiring CA information for follower catalog manager: success
I20260621 02:53:23.034371 25397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14ef3da058e84bf0a2f96540fb47e143: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:53:23.128747 25419 catalog_manager.cc:1259] Loaded cluster ID: 4f988c5b3fc04683b3b551f988c83e09
I20260621 02:53:23.128856 25419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: loading cluster ID for follower catalog manager: success
I20260621 02:53:23.131582 25419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: acquiring CA information for follower catalog manager: success
I20260621 02:53:23.133374 25419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ead181889ad4168bfa62345beae5cc3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:53:23.226768 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25141
I20260621 02:53:23.228129 25203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:23.337575 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25141
I20260621 02:53:23.366073 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25215
I20260621 02:53:23.367163 25277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:23.476644 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25215
I20260621 02:53:23.495324 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25289
I20260621 02:53:23.496436 25351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:23.604279 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25289
2026-06-21T02:53:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4584 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:53:23 dist-test-slave-bc52 krb5kdc[25430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:53:23 dist-test-slave-bc52 krb5kdc[25430](info): set up 2 sockets
Jun 21 02:53:23 dist-test-slave-bc52 krb5kdc[25430](info): commencing operation
krb5kdc: starting...
W20260621 02:53:24.717533 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:53:24 dist-test-slave-bc52 krb5kdc[25430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:53:24.928764 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:40133
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:25.242566 25443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:25.242892 25443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:25.243045 25443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:25.250098 25443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:25.250226 25443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:25.250326 25443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:25.250442 25443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:25.250535 25443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:25.259919 25443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:25.261143 25443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:25.262389 25443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:25.273716 25451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:25.273782 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:25.273759 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:25.274439 25443 server_base.cc:1034] running on GCE node
I20260621 02:53:25.275099 25443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:25.276363 25443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:25.277558 25443 hybrid_clock.cc:648] HybridClock initialized: now 1782010405277522 us; error 27 us; skew 500 ppm
I20260621 02:53:25.277863 25443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:25 dist-test-slave-bc52 krb5kdc[25430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010405, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:25.283912 25443 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:25.285499 25443 webserver.cc:460] Webserver started at http://127.11.109.62:39189/ using document root <none> and password file <none>
I20260621 02:53:25.286221 25443 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:25.286353 25443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:25.286666 25443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:25.289654 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "3a97380edb4a4c5abb2c3c9ad87b808b"
format_stamp: "Formatted at 2026-06-21 02:53:25 on dist-test-slave-bc52"
server_key: "530c41b7a2187cd33746c43f2b2d76cb"
server_key_iv: "3d1bc6d1f27577fb23ec3a48208719ba"
server_key_version: "encryptionkey@0"
I20260621 02:53:25.290387 25443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "3a97380edb4a4c5abb2c3c9ad87b808b"
format_stamp: "Formatted at 2026-06-21 02:53:25 on dist-test-slave-bc52"
server_key: "530c41b7a2187cd33746c43f2b2d76cb"
server_key_iv: "3d1bc6d1f27577fb23ec3a48208719ba"
server_key_version: "encryptionkey@0"
I20260621 02:53:25.295171 25443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:53:25.298405 25458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:25.299615 25443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:53:25.299788 25443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "3a97380edb4a4c5abb2c3c9ad87b808b"
format_stamp: "Formatted at 2026-06-21 02:53:25 on dist-test-slave-bc52"
server_key: "530c41b7a2187cd33746c43f2b2d76cb"
server_key_iv: "3d1bc6d1f27577fb23ec3a48208719ba"
server_key_version: "encryptionkey@0"
I20260621 02:53:25.299993 25443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:25.315275 25443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:25.319460 25443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:25.319734 25443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:25.338419 25443 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:40133
I20260621 02:53:25.338438 25509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:40133 every 8 connection(s)
I20260621 02:53:25.339764 25443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:25.343386 25510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:25.348121 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25443
I20260621 02:53:25.348376 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:53:25.349116 11700 external_mini_cluster.cc:1428] Setting key 79266b9d883256f91d6cee1501075ce1
I20260621 02:53:25.353931 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b: Bootstrap starting.
I20260621 02:53:25.357520 25510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:25.358714 25510 log.cc:824] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:25.361430 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b: No bootstrap required, opened a new log
Jun 21 02:53:25 dist-test-slave-bc52 krb5kdc[25430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:25.369380 25510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 40133 } }
I20260621 02:53:25.369728 25510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:25.369845 25510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a97380edb4a4c5abb2c3c9ad87b808b, State: Initialized, Role: FOLLOWER
I20260621 02:53:25.370445 25510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 40133 } }
I20260621 02:53:25.370702 25510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:25.370874 25510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:25.371061 25510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:25.372181 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:25.351182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.11.109.62:40133)
0621 02:53:25.352004 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:25.352015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:25.352058 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:53:25.353733 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:25.353754 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:25.354139 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:25.354459 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:25.356358 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:25.357742 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:25.359576 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:25.360207 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:25.360392 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:25.364891 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:25.364898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:25.364912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:25.364944 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:25.368120 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:25.368898 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:25.368906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:25.368919 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:25.369081 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:25.369476 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:25.369482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:25.369485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:25.370088 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:25.370408 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:25.370805 (+ 397us) server_negotiation.cc:299] Negotiation successful
0621 02:53:25.371223 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":397,"threads_started":1}
I20260621 02:53:25.373485 25510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 40133 } }
I20260621 02:53:25.374033 25510 leader_election.cc:302] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a97380edb4a4c5abb2c3c9ad87b808b; no voters:
I20260621 02:53:25.374737 25510 leader_election.cc:288] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:53:25.374825 25515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:25.376247 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:53:25.376343 25515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [term 1 LEADER]: Becoming Leader. State: Replica: 3a97380edb4a4c5abb2c3c9ad87b808b, State: Running, Role: LEADER
I20260621 02:53:25.376863 25515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 40133 } }
I20260621 02:53:25.377009 25510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:25.380921 25517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a97380edb4a4c5abb2c3c9ad87b808b. Latest consensus state: current_term: 1 leader_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 40133 } } }
I20260621 02:53:25.381433 25517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:25.382664 25516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a97380edb4a4c5abb2c3c9ad87b808b" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 40133 } } }
I20260621 02:53:25.382946 25516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:25.385680 25522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:25.390631 25522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:25.400141 25522 catalog_manager.cc:1347] Generated new cluster ID: 05b671ecb8894240bbbbf5f7802e9afc
I20260621 02:53:25.400295 25522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:25.416867 25522 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:53:25.418170 25522 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:25.426229 25522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a97380edb4a4c5abb2c3c9ad87b808b: Generated new TSK 0
I20260621 02:53:25.427055 25522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:53:25.437961 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25443
I20260621 02:53:25.439067 25505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:25.528398 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25443
2026-06-21T02:53:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1919 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:53:25 dist-test-slave-bc52 krb5kdc[25538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:53:25 dist-test-slave-bc52 krb5kdc[25538](info): set up 2 sockets
Jun 21 02:53:25 dist-test-slave-bc52 krb5kdc[25538](info): commencing operation
krb5kdc: starting...
W20260621 02:53:26.609807 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.009s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:53:26 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:53:26.830336 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45657
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:27.148155 25551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:27.148452 25551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:27.148589 25551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:27.155473 25551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:27.155570 25551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:27.155616 25551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:27.155666 25551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:27.155710 25551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:27.164968 25551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:27.166177 25551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:27.167339 25551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:27.178623 25557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:27.178594 25559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:27.178841 25556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:27.179219 25551 server_base.cc:1034] running on GCE node
I20260621 02:53:27.179970 25551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:27.181282 25551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:27.182512 25551 hybrid_clock.cc:648] HybridClock initialized: now 1782010407182471 us; error 51 us; skew 500 ppm
I20260621 02:53:27.182850 25551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:27 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010407, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:27.185817 25551 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:27.187227 25551 webserver.cc:460] Webserver started at http://127.11.109.62:33883/ using document root <none> and password file <none>
I20260621 02:53:27.187880 25551 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:27.188023 25551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:27.188308 25551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:27.191264 25551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "85386363d27e4eb0a652ab3dcdd640fa"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "d4e6834fdace63f7f4fb26a7e22969ea"
server_key_iv: "4ce88ed67851cd5016a00f0f6efb91c5"
server_key_version: "encryptionkey@0"
I20260621 02:53:27.191999 25551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "85386363d27e4eb0a652ab3dcdd640fa"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "d4e6834fdace63f7f4fb26a7e22969ea"
server_key_iv: "4ce88ed67851cd5016a00f0f6efb91c5"
server_key_version: "encryptionkey@0"
I20260621 02:53:27.196741 25551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260621 02:53:27.199963 25566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:27.201097 25551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:53:27.201308 25551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "85386363d27e4eb0a652ab3dcdd640fa"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "d4e6834fdace63f7f4fb26a7e22969ea"
server_key_iv: "4ce88ed67851cd5016a00f0f6efb91c5"
server_key_version: "encryptionkey@0"
I20260621 02:53:27.201494 25551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:27.218309 25551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:27.221967 25551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:27.222221 25551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:27.239703 25551 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45657
I20260621 02:53:27.239719 25617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45657 every 8 connection(s)
I20260621 02:53:27.240939 25551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:27.244287 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:27.250167 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25551
I20260621 02:53:27.250396 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:53:27.251001 11700 external_mini_cluster.cc:1428] Setting key fecca965f0e449ddded10c8dc80343c0
I20260621 02:53:27.255398 25618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap starting.
I20260621 02:53:27.258287 25618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:27.259256 25618 log.cc:824] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:27.262188 25618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: No bootstrap required, opened a new log
Jun 21 02:53:27 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:27.271924 25618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:27.272372 25618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:27.272506 25618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85386363d27e4eb0a652ab3dcdd640fa, State: Initialized, Role: FOLLOWER
I20260621 02:53:27.273243 25618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:27.273581 25618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:27.273761 25618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:27.273981 25618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:27.274788 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:27.252904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.11.109.62:45657)
0621 02:53:27.253658 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:27.253670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:27.253707 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:53:27.255282 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:27.255303 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:27.255700 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:27.256055 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:27.257456 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:27.258810 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:27.260445 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:27.261438 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:27.261617 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:27.265933 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:27.265944 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:27.265962 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:27.266006 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:27.270064 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:27.270793 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:27.270802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:27.270811 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:27.271032 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:27.271394 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:27.271416 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:27.271418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:27.272106 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:27.272414 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:27.272957 (+ 543us) server_negotiation.cc:299] Negotiation successful
0621 02:53:27.273580 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":356,"threads_started":1}
I20260621 02:53:27.276530 25618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:27.277019 25618 leader_election.cc:302] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85386363d27e4eb0a652ab3dcdd640fa; no voters:
I20260621 02:53:27.277726 25618 leader_election.cc:288] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:53:27.277846 25623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:27.278540 25623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 LEADER]: Becoming Leader. State: Replica: 85386363d27e4eb0a652ab3dcdd640fa, State: Running, Role: LEADER
I20260621 02:53:27.279129 25623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:27.279903 25618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:27.281994 25625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85386363d27e4eb0a652ab3dcdd640fa. Latest consensus state: current_term: 1 leader_uuid: "85386363d27e4eb0a652ab3dcdd640fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } } }
I20260621 02:53:27.282328 25625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:27.282030 25624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85386363d27e4eb0a652ab3dcdd640fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } } }
I20260621 02:53:27.282433 25624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:27.284793 25629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:27.290438 25629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:27.298614 25629 catalog_manager.cc:1347] Generated new cluster ID: 0ab291d5ec004a6aa56a02096465e66b
I20260621 02:53:27.298725 25629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:27.329231 25629 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:53:27.330633 25629 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:27.338629 25629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Generated new TSK 0
I20260621 02:53:27.339422 25629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:53:27.400743 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45657
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:27.725090 25644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:27.725521 25644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:27.725696 25644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:27.732796 25644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:27.732957 25644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:27.733126 25644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:53:27.742268 25644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:27.743533 25644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:27.744954 25644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:27.758261 25649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:27.758175 25652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:27.758224 25650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:27.758987 25644 server_base.cc:1034] running on GCE node
I20260621 02:53:27.759640 25644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:27.760510 25644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:27.761734 25644 hybrid_clock.cc:648] HybridClock initialized: now 1782010407761685 us; error 44 us; skew 500 ppm
I20260621 02:53:27.762049 25644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:27 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010407, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:27.765123 25644 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:53:27.766595 25644 webserver.cc:460] Webserver started at http://127.11.109.1:37149/ using document root <none> and password file <none>
I20260621 02:53:27.767256 25644 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:27.767397 25644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:27.767685 25644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:27.770558 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "f55824adbd5645c5a14f5afa81f9c7bc"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "272c8f4411dfc822424c98e422f76ca4"
server_key_iv: "11e66ec616893a8293700dea076388ec"
server_key_version: "encryptionkey@0"
I20260621 02:53:27.771341 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "f55824adbd5645c5a14f5afa81f9c7bc"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "272c8f4411dfc822424c98e422f76ca4"
server_key_iv: "11e66ec616893a8293700dea076388ec"
server_key_version: "encryptionkey@0"
I20260621 02:53:27.776268 25644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260621 02:53:27.779632 25659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:27.781455 25644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:53:27.781620 25644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f55824adbd5645c5a14f5afa81f9c7bc"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "272c8f4411dfc822424c98e422f76ca4"
server_key_iv: "11e66ec616893a8293700dea076388ec"
server_key_version: "encryptionkey@0"
I20260621 02:53:27.781810 25644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:27.806531 25644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:27.810272 25644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:27.810555 25644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:27.811528 25644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:27.812865 25644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:27.812971 25644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:27.813133 25644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:27.813226 25644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 02:53:27.846083 25644 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:45117
I20260621 02:53:27.846166 25771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:45117 every 8 connection(s)
I20260621 02:53:27.847394 25644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:53:27.853542 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25644
I20260621 02:53:27.853749 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:53:27.854403 11700 external_mini_cluster.cc:1428] Setting key 0d06a56e3bf5e2086866b2ce08dd468e
Jun 21 02:53:27 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010407, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:27.868592 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:27.849921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:57893 (local address 127.11.109.62:45657)
0621 02:53:27.850340 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:27.850345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:27.851335 (+ 990us) server_negotiation.cc:407] Connection header received
0621 02:53:27.853145 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:27.853151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:27.853280 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:27.853400 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:27.854976 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:27.855838 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:27.857644 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:27.858418 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:27.858563 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:27.862348 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:27.862353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:27.862356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:27.862394 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:27.865404 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:27.866104 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:27.866113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:27.866120 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:27.866263 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:27.866669 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:27.866676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:27.866682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:27.867045 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:27.867203 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:27.867839 (+ 636us) server_negotiation.cc:299] Negotiation successful
0621 02:53:27.868103 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260621 02:53:27.869242 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:27.850498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:57893)
0621 02:53:27.851183 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:27.851220 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:27.852860 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:27.853628 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:27.853636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:27.854212 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:27.854814 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:27.854833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:27.856040 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:27.856047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:27.857494 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:27.857506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:27.858619 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:27.858623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:27.858814 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:27.859499 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:27.859519 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:27.862150 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:27.865547 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:27.865554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:27.865565 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:27.865986 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:27.866374 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:27.866377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:27.866380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:27.866562 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:27.867172 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:27.867177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:27.867472 (+ 295us) client_negotiation.cc:769] Sending connection context
0621 02:53:27.867696 (+ 224us) client_negotiation.cc:240] Negotiation successful
0621 02:53:27.868080 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":279,"threads_started":1}
I20260621 02:53:27.870981 25772 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45657
I20260621 02:53:27.871313 25772 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:27.872020 25772 heartbeater.cc:511] Master 127.11.109.62:45657 requested a full tablet report, sending...
I20260621 02:53:27.874307 25583 ts_manager.cc:194] Registered new tserver with Master: f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117)
I20260621 02:53:27.875759 25583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:57893
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:53:27.905053 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45657
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:28.228225 25778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:28.228543 25778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:28.228722 25778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:28.236284 25778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:28.236433 25778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:28.236621 25778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:53:28.250751 25778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:28.252267 25778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:28.254010 25778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:28.268529 25784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:28.268587 25786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:28.268566 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:28.274624 25778 server_base.cc:1034] running on GCE node
I20260621 02:53:28.275178 25778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:28.276037 25778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:28.277257 25778 hybrid_clock.cc:648] HybridClock initialized: now 1782010408277217 us; error 31 us; skew 500 ppm
I20260621 02:53:28.277518 25778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:28 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:28.280540 25778 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:53:28.282037 25778 webserver.cc:460] Webserver started at http://127.11.109.2:34007/ using document root <none> and password file <none>
I20260621 02:53:28.282701 25778 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:28.282817 25778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:28.283094 25778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:28.286090 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "44bc2988640b4c719d5b99cfa9601bcc"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "2f390a2774f3926bbee2c1ec69ed5dbc"
server_key_iv: "9898763847801ea630e4543b4e786889"
server_key_version: "encryptionkey@0"
I20260621 02:53:28.286820 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "44bc2988640b4c719d5b99cfa9601bcc"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "2f390a2774f3926bbee2c1ec69ed5dbc"
server_key_iv: "9898763847801ea630e4543b4e786889"
server_key_version: "encryptionkey@0"
I20260621 02:53:28.291793 25778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:53:28.295127 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:28.296434 25778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:53:28.296593 25778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "44bc2988640b4c719d5b99cfa9601bcc"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "2f390a2774f3926bbee2c1ec69ed5dbc"
server_key_iv: "9898763847801ea630e4543b4e786889"
server_key_version: "encryptionkey@0"
I20260621 02:53:28.296794 25778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:28.312178 25778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:28.315972 25778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:28.316217 25778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:28.317092 25778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:28.318322 25778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:28.318403 25778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:28.318562 25778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:28.318615 25778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:28.352458 25778 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:33029
I20260621 02:53:28.352507 25905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:33029 every 8 connection(s)
I20260621 02:53:28.353777 25778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:53:28.356025 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25778
I20260621 02:53:28.356256 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:53:28.356702 11700 external_mini_cluster.cc:1428] Setting key 0513200d5ed9b84194c8ebc643c77796
Jun 21 02:53:28 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:28.377684 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.356339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:53011 (local address 127.11.109.62:45657)
0621 02:53:28.356535 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:28.356540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:28.357936 (+ 1396us) server_negotiation.cc:407] Connection header received
0621 02:53:28.360417 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:28.360421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:28.360483 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:28.360565 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:28.362636 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.363332 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.365899 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.366534 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.366658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.370644 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:28.370653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:28.370660 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:28.370705 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:28.374369 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:28.375108 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:28.375117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:28.375123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:28.375295 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:28.375739 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:28.375747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:28.375752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:28.376128 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:28.376296 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:28.376929 (+ 633us) server_negotiation.cc:299] Negotiation successful
0621 02:53:28.377222 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:53:28.378362 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.356949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:53011)
0621 02:53:28.357796 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:28.357833 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:28.360119 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:28.360855 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:28.360869 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:28.361660 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:28.362467 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.362494 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.363502 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.363510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.365736 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.365753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.366780 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.366789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.367105 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.367810 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:28.367833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:28.370379 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:28.374522 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:28.374528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:28.374541 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:28.374972 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:28.375422 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:28.375426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:28.375428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:28.375613 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:28.376246 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:28.376252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:28.376554 (+ 302us) client_negotiation.cc:769] Sending connection context
0621 02:53:28.376774 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 02:53:28.377152 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":351,"threads_started":1}
I20260621 02:53:28.379952 25906 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45657
I20260621 02:53:28.380255 25906 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:28.381037 25906 heartbeater.cc:511] Master 127.11.109.62:45657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
I20260621 02:53:28.382589 25583 ts_manager.cc:194] Registered new tserver with Master: 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
I20260621 02:53:28.383414 25583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:53011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:53:28.410562 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45657
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:28.737497 25911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:28.737793 25911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:28.737963 25911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:28.744973 25911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:28.745119 25911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:28.745311 25911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:53:28.754356 25911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:28.755488 25911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:28.756726 25911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:28.769336 25916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:28.769300 25919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:28.769368 25917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:28.770143 25911 server_base.cc:1034] running on GCE node
I20260621 02:53:28.770627 25911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:28.771406 25911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:28.772588 25911 hybrid_clock.cc:648] HybridClock initialized: now 1782010408772535 us; error 52 us; skew 500 ppm
I20260621 02:53:28.772848 25911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:28 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:28.775925 25911 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:53:28.777362 25911 webserver.cc:460] Webserver started at http://127.11.109.3:41081/ using document root <none> and password file <none>
I20260621 02:53:28.777995 25911 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:28.778124 25911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:28.778415 25911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:28.781375 25911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "a457d954efb14a6c937bdeff7d4536d3"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "be0a1366f9de2fb9004d016a7c81135f"
server_key_iv: "ca2b1aa0511ca1a28ea96d34829e37d0"
server_key_version: "encryptionkey@0"
I20260621 02:53:28.782126 25911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "a457d954efb14a6c937bdeff7d4536d3"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "be0a1366f9de2fb9004d016a7c81135f"
server_key_iv: "ca2b1aa0511ca1a28ea96d34829e37d0"
server_key_version: "encryptionkey@0"
I20260621 02:53:28.787101 25911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260621 02:53:28.790376 25926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:28.791569 25911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:53:28.791733 25911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "a457d954efb14a6c937bdeff7d4536d3"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "be0a1366f9de2fb9004d016a7c81135f"
server_key_iv: "ca2b1aa0511ca1a28ea96d34829e37d0"
server_key_version: "encryptionkey@0"
I20260621 02:53:28.791927 25911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:28.814929 25911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:28.819451 25911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:28.819749 25911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:28.820700 25911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:28.822029 25911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:28.822134 25911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:28.822276 25911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:28.822352 25911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:28.855626 25911 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35051
I20260621 02:53:28.855656 26038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35051 every 8 connection(s)
I20260621 02:53:28.856958 25911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:53:28.860668 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 25911
I20260621 02:53:28.860872 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:53:28.861485 11700 external_mini_cluster.cc:1428] Setting key 9420394cd3f405932a672b4056ab3975
Jun 21 02:53:28 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010408, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:28.877334 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.859551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:49213 (local address 127.11.109.62:45657)
0621 02:53:28.859776 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:28.859782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:28.861070 (+ 1288us) server_negotiation.cc:407] Connection header received
0621 02:53:28.863533 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:28.863539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:28.863643 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:28.863759 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:28.865344 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.866104 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.867597 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.868000 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.868134 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.871453 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:28.871458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:28.871461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:28.871492 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:28.874018 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:28.874705 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:28.874708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:28.874711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:28.874843 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:28.875611 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:28.875618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:28.875625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:28.875898 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:28.876036 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:28.876882 (+ 846us) server_negotiation.cc:299] Negotiation successful
0621 02:53:28.877048 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 02:53:28.878227 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.860029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:49213)
0621 02:53:28.860894 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:28.860951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:28.863223 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:28.863996 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:28.864003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:28.864607 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:28.865132 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.865149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.866252 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.866257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.867459 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.867469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.868220 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.868231 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.868467 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.869131 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:28.869152 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:28.871275 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:28.874134 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:28.874140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:28.874150 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:28.874553 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:28.875142 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:28.875146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:28.875150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:28.875388 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:28.876035 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:28.876049 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:28.876454 (+ 405us) client_negotiation.cc:769] Sending connection context
0621 02:53:28.876709 (+ 255us) client_negotiation.cc:240] Negotiation successful
0621 02:53:28.877155 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":314,"threads_started":1}
I20260621 02:53:28.878664 25772 heartbeater.cc:503] Master 127.11.109.62:45657 was elected leader, sending a full tablet report...
I20260621 02:53:28.879637 26039 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45657
I20260621 02:53:28.879918 26039 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:28.880633 26039 heartbeater.cc:511] Master 127.11.109.62:45657 requested a full tablet report, sending...
I20260621 02:53:28.881997 25583 ts_manager.cc:194] Registered new tserver with Master: a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051)
I20260621 02:53:28.882767 25583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:49213
I20260621 02:53:28.888872 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:53:28.905385 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.893346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.11.109.62:45657)
0621 02:53:28.893547 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:28.893557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:28.893926 (+ 369us) server_negotiation.cc:407] Connection header received
0621 02:53:28.894046 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:28.894050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:28.894107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:28.894180 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:28.895477 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.896215 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.897568 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.897971 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.898067 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.900379 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:28.900384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:28.900388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:28.900418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:28.902987 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:28.903602 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:28.903606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:28.903608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:28.903727 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:28.904041 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:28.904044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:28.904046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:28.904395 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:28.904594 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:28.904870 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 02:53:28.905037 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:53:28.909334 25583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:53:28.916522 25583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:53:28.938182 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.927579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:33029 (local address 127.0.0.1:43438)
0621 02:53:28.928458 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:28.928503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:28.928713 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:28.930419 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:28.930426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:28.930455 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:28.930844 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.930856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.932854 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.932859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.935209 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.935223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.937087 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.937093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.937316 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.937369 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 02:53:28.937528 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 02:53:28.937715 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":2,"thread_start_us":412,"threads_started":1}
I20260621 02:53:28.938832 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.927781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:45240)
0621 02:53:28.930190 (+ 2409us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:28.930207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:28.930345 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:28.931209 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:28.931213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:28.931236 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:28.931949 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.931961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.933820 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.933825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.935870 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.935884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.938017 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.938023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.938243 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.938276 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:53:28.938377 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:53:28.938503 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2302,"thread_start_us":2440,"threads_started":1}
I20260621 02:53:28.939368 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.927706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.11.109.2:33029)
0621 02:53:28.929777 (+ 2071us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:28.929785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:28.929810 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:53:28.929920 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:28.929931 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:28.930190 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:28.931015 (+ 825us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:28.931205 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.932673 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.935407 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.936836 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.938643 (+ 1807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.938838 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:28.938910 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 02:53:28.939086 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1913,"thread_start_us":1990,"threads_started":1}
I20260621 02:53:28.939693 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.928309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.11.109.1:45117)
0621 02:53:28.929367 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:28.929375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:28.930467 (+ 1092us) server_negotiation.cc:407] Connection header received
0621 02:53:28.930572 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:28.930578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:28.930792 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:28.930990 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:28.932164 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.933658 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.936060 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.937538 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.937656 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.937872 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:28.939215 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0621 02:53:28.939380 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":748,"threads_started":1}
I20260621 02:53:28.943687 25841 tablet_service.cc:1467] Processing CreateTablet for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 (DEFAULT_TABLE table=test-table [id=ebaffc7fbebd4ed1a0e79fccc20e598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:28.943687 25707 tablet_service.cc:1467] Processing CreateTablet for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 (DEFAULT_TABLE table=test-table [id=ebaffc7fbebd4ed1a0e79fccc20e598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:28.945101 25841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:28.945411 25707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:28.945839 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.929655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35051 (local address 127.0.0.1:40038)
0621 02:53:28.938396 (+ 8741us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:28.938412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:28.938489 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:28.940693 (+ 2204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:28.940697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:28.940718 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:28.940881 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.940893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.942267 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.942275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.943439 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:28.943451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.945027 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.945033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:28.945236 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.945293 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 02:53:28.945411 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 02:53:28.945544 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8639,"thread_start_us":10352,"threads_started":1}
I20260621 02:53:28.945935 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:28.929947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.11.109.3:35051)
0621 02:53:28.940157 (+ 10210us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:28.940167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:28.940200 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:53:28.940284 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:28.940289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:28.940525 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:28.940716 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:28.941001 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.942124 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.943588 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:28.944884 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:28.945017 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:28.945261 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:28.945451 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:53:28.945600 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10031,"thread_start_us":12310,"threads_started":1}
I20260621 02:53:28.949002 25974 tablet_service.cc:1467] Processing CreateTablet for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 (DEFAULT_TABLE table=test-table [id=ebaffc7fbebd4ed1a0e79fccc20e598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:28.950177 25974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:28.959790 26058 tablet_bootstrap.cc:492] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Bootstrap starting.
I20260621 02:53:28.960527 26057 tablet_bootstrap.cc:492] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Bootstrap starting.
I20260621 02:53:28.962666 26059 tablet_bootstrap.cc:492] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Bootstrap starting.
I20260621 02:53:28.964960 26057 tablet_bootstrap.cc:654] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:28.965078 26058 tablet_bootstrap.cc:654] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:28.966077 26059 tablet_bootstrap.cc:654] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:28.966090 26058 log.cc:824] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:28.966254 26057 log.cc:824] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:28.967252 26059 log.cc:824] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:28.969041 26058 tablet_bootstrap.cc:492] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: No bootstrap required, opened a new log
I20260621 02:53:28.969338 26058 ts_tablet_manager.cc:1397] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:53:28.969928 26057 tablet_bootstrap.cc:492] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: No bootstrap required, opened a new log
I20260621 02:53:28.970178 26057 ts_tablet_manager.cc:1397] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:53:28.975980 26058 raft_consensus.cc:348] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:28.976361 26058 raft_consensus.cc:374] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:28.976472 26058 raft_consensus.cc:729] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bc2988640b4c719d5b99cfa9601bcc, State: Initialized, Role: FOLLOWER
I20260621 02:53:28.976778 26057 raft_consensus.cc:348] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:28.977205 26057 raft_consensus.cc:374] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:28.977319 26057 raft_consensus.cc:729] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55824adbd5645c5a14f5afa81f9c7bc, State: Initialized, Role: FOLLOWER
I20260621 02:53:28.977190 26058 consensus_queue.cc:260] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:28.978039 26057 consensus_queue.cc:260] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:28.978861 26059 tablet_bootstrap.cc:492] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: No bootstrap required, opened a new log
I20260621 02:53:28.978871 25906 heartbeater.cc:503] Master 127.11.109.62:45657 was elected leader, sending a full tablet report...
I20260621 02:53:28.979245 26059 ts_tablet_manager.cc:1397] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260621 02:53:28.979535 26058 ts_tablet_manager.cc:1428] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260621 02:53:28.981194 26057 ts_tablet_manager.cc:1428] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260621 02:53:28.988517 26059 raft_consensus.cc:348] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:28.988850 26059 raft_consensus.cc:374] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:28.988965 26059 raft_consensus.cc:729] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a457d954efb14a6c937bdeff7d4536d3, State: Initialized, Role: FOLLOWER
I20260621 02:53:28.989565 26059 consensus_queue.cc:260] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:28.990465 26039 heartbeater.cc:503] Master 127.11.109.62:45657 was elected leader, sending a full tablet report...
I20260621 02:53:28.991029 26059 ts_tablet_manager.cc:1428] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260621 02:53:29.095180 26065 raft_consensus.cc:482] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:29.095450 26065 raft_consensus.cc:504] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:29.097293 26065 leader_election.cc:288] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029), f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117)
W20260621 02:53:29.099777 25773 tablet.cc:2369] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:29.103607 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.097628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:54555 (local address 127.11.109.2:33029)
0621 02:53:29.097893 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:29.097900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:29.098167 (+ 267us) server_negotiation.cc:407] Connection header received
0621 02:53:29.098405 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:29.098413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:29.098506 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:29.098609 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:29.099021 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.100108 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.101945 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.102702 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.102869 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.102942 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:29.103203 (+ 261us) server_negotiation.cc:299] Negotiation successful
0621 02:53:29.103348 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:53:29.103613 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.097683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:33029 (local address 127.11.109.3:54555)
0621 02:53:29.098062 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:29.098084 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:29.098223 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:29.098640 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:29.098643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:29.098666 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:29.098849 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:29.098858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.100275 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.100280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:29.101794 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:29.101806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.102855 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.102860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:29.103047 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.103070 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:53:29.103177 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 02:53:29.103326 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260621 02:53:29.104125 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.097709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:34997 (local address 127.11.109.1:45117)
0621 02:53:29.097920 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:29.097928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:29.098862 (+ 934us) server_negotiation.cc:407] Connection header received
0621 02:53:29.098997 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:29.099006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:29.099065 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:29.099132 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:29.099743 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.100769 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.102429 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.103200 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.103402 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.103471 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:29.103743 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 02:53:29.103886 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:53:29.104148 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.097966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.11.109.3:34997)
0621 02:53:29.098740 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:29.098756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:29.098872 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:29.099170 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:29.099174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:29.099197 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:29.099555 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:29.099566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.100932 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.100943 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:29.102278 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:29.102289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.103422 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.103428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:29.103584 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.103609 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 02:53:29.103708 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:53:29.103823 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":422,"threads_started":1}
I20260621 02:53:29.104908 25861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269e2e8ddf81403a9a0f6ddd60c63ab9" candidate_uuid: "a457d954efb14a6c937bdeff7d4536d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bc2988640b4c719d5b99cfa9601bcc" is_pre_election: true
I20260621 02:53:29.105177 25727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269e2e8ddf81403a9a0f6ddd60c63ab9" candidate_uuid: "a457d954efb14a6c937bdeff7d4536d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" is_pre_election: true
I20260621 02:53:29.105440 25861 raft_consensus.cc:2454] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a457d954efb14a6c937bdeff7d4536d3 in term 0.
I20260621 02:53:29.105785 25727 raft_consensus.cc:2454] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a457d954efb14a6c937bdeff7d4536d3 in term 0.
I20260621 02:53:29.106292 25930 leader_election.cc:302] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44bc2988640b4c719d5b99cfa9601bcc, a457d954efb14a6c937bdeff7d4536d3; no voters:
I20260621 02:53:29.106729 26065 raft_consensus.cc:2781] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:53:29.106886 26065 raft_consensus.cc:482] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260621 02:53:29.106887 25907 tablet.cc:2369] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:29.106985 26065 raft_consensus.cc:3037] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Advancing to term 1
W20260621 02:53:29.108565 26040 tablet.cc:2369] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:29.109905 26065 raft_consensus.cc:504] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:29.110414 26065 leader_election.cc:288] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [CANDIDATE]: Term 1 election: Requested vote from peers 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029), f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117)
I20260621 02:53:29.110879 25861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269e2e8ddf81403a9a0f6ddd60c63ab9" candidate_uuid: "a457d954efb14a6c937bdeff7d4536d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bc2988640b4c719d5b99cfa9601bcc"
I20260621 02:53:29.110893 25727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269e2e8ddf81403a9a0f6ddd60c63ab9" candidate_uuid: "a457d954efb14a6c937bdeff7d4536d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f55824adbd5645c5a14f5afa81f9c7bc"
I20260621 02:53:29.111071 25861 raft_consensus.cc:3037] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:29.111092 25727 raft_consensus.cc:3037] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:29.113134 25727 raft_consensus.cc:2454] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a457d954efb14a6c937bdeff7d4536d3 in term 1.
I20260621 02:53:29.113157 25861 raft_consensus.cc:2454] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a457d954efb14a6c937bdeff7d4536d3 in term 1.
I20260621 02:53:29.113587 25929 leader_election.cc:302] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a457d954efb14a6c937bdeff7d4536d3, f55824adbd5645c5a14f5afa81f9c7bc; no voters:
I20260621 02:53:29.113886 26065 raft_consensus.cc:2781] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:29.114337 26065 raft_consensus.cc:686] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 1 LEADER]: Becoming Leader. State: Replica: a457d954efb14a6c937bdeff7d4536d3, State: Running, Role: LEADER
I20260621 02:53:29.114766 26065 consensus_queue.cc:237] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } }
I20260621 02:53:29.119427 25583 catalog_manager.cc:5495] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "a457d954efb14a6c937bdeff7d4536d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } health_report { overall_health: HEALTHY } } }
I20260621 02:53:29.138029 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.132863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.11.109.3:35051)
0621 02:53:29.133065 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:29.133069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:29.133085 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:53:29.133285 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:29.133288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:29.133376 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:29.133517 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:53:29.133833 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.134508 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.135998 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.136388 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.136478 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.136901 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:53:29.137466 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:53:29.137569 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:29.137664 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 02:53:29.137738 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:53:29.145105 25727 raft_consensus.cc:1261] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Refusing update from remote peer a457d954efb14a6c937bdeff7d4536d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:29.145152 25861 raft_consensus.cc:1261] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Refusing update from remote peer a457d954efb14a6c937bdeff7d4536d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:29.146040 26067 consensus_queue.cc:1035] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:29.146337 26065 consensus_queue.cc:1035] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:29.157924 26070 mvcc.cc:205] Tried to move back new op lower bound from 7299114635850014720 to 7299114635737595904. Current Snapshot: MvccSnapshot[applied={T|T < 7299114635850014720}]
I20260621 02:53:29.159713 26072 mvcc.cc:205] Tried to move back new op lower bound from 7299114635850014720 to 7299114635737595904. Current Snapshot: MvccSnapshot[applied={T|T < 7299114635850014720}]
I20260621 02:53:29.174995 25582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:29.175278 25582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:29.178982 25582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Sending DeleteTablet for 3 replicas of tablet 269e2e8ddf81403a9a0f6ddd60c63ab9
I20260621 02:53:29.180258 25841 tablet_service.cc:1514] Processing DeleteTablet for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:29 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:43438
I20260621 02:53:29.180275 25707 tablet_service.cc:1514] Processing DeleteTablet for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:29 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:45240
I20260621 02:53:29.180342 25974 tablet_service.cc:1514] Processing DeleteTablet for tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:29 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:40038
I20260621 02:53:29.181653 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25644
I20260621 02:53:29.194218 26080 tablet_replica.cc:330] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: stopping tablet replica
I20260621 02:53:29.194486 26082 tablet_replica.cc:330] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: stopping tablet replica
I20260621 02:53:29.194693 26080 raft_consensus.cc:2229] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:29.195055 26082 raft_consensus.cc:2229] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:53:29.195267 26080 raft_consensus.cc:2258] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:29.195845 26082 raft_consensus.cc:2258] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:29.197111 26081 tablet_replica.cc:330] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: stopping tablet replica
I20260621 02:53:29.197541 26081 raft_consensus.cc:2229] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:29.198026 26081 raft_consensus.cc:2258] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:29.198271 26082 ts_tablet_manager.cc:1905] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:29.198947 26080 ts_tablet_manager.cc:1905] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:29.199934 26081 ts_tablet_manager.cc:1905] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:29.206265 26082 ts_tablet_manager.cc:1918] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:29.206509 26082 log.cc:1196] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/269e2e8ddf81403a9a0f6ddd60c63ab9
I20260621 02:53:29.206867 26081 ts_tablet_manager.cc:1918] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:29.207043 26081 log.cc:1196] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/269e2e8ddf81403a9a0f6ddd60c63ab9
I20260621 02:53:29.207278 26082 ts_tablet_manager.cc:1939] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P a457d954efb14a6c937bdeff7d4536d3: Deleting consensus metadata
I20260621 02:53:29.207577 26081 ts_tablet_manager.cc:1939] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P f55824adbd5645c5a14f5afa81f9c7bc: Deleting consensus metadata
I20260621 02:53:29.209079 25570 catalog_manager.cc:4841] TS a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 (table test-table [id=ebaffc7fbebd4ed1a0e79fccc20e598d]) successfully deleted
I20260621 02:53:29.209276 25569 catalog_manager.cc:4841] TS f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117): tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 (table test-table [id=ebaffc7fbebd4ed1a0e79fccc20e598d]) successfully deleted
I20260621 02:53:29.209656 26080 ts_tablet_manager.cc:1918] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:29.209820 26080 log.cc:1196] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/269e2e8ddf81403a9a0f6ddd60c63ab9
I20260621 02:53:29.210246 26080 ts_tablet_manager.cc:1939] T 269e2e8ddf81403a9a0f6ddd60c63ab9 P 44bc2988640b4c719d5b99cfa9601bcc: Deleting consensus metadata
I20260621 02:53:29.211534 25570 catalog_manager.cc:4841] TS 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): tablet 269e2e8ddf81403a9a0f6ddd60c63ab9 (table test-table [id=ebaffc7fbebd4ed1a0e79fccc20e598d]) successfully deleted
Jun 21 02:53:29 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:53:29.219833 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.183108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.11.109.1:45117)
0621 02:53:29.183347 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:29.183356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:29.189782 (+ 6426us) server_negotiation.cc:407] Connection header received
0621 02:53:29.190866 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:29.190874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:29.190989 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:29.191078 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:29.196307 (+ 5229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.197597 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.204924 (+ 7327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.205632 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.205727 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.213522 (+ 7795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:29.213536 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:29.213553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:29.213599 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:29.217026 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:29.217773 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:29.217782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:29.217786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:29.217957 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:29.218305 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:29.218310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:29.218314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:29.218733 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:29.219029 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:29.219228 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 02:53:29.219454 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:53:29.220510 25767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:29.324394 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25644
I20260621 02:53:29.342103 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25778
Jun 21 02:53:29 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:53:29.357928 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.342885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.11.109.2:33029)
0621 02:53:29.343079 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:29.343085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:29.343107 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:53:29.343179 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:29.343184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:29.343266 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:29.343396 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:29.344552 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.345687 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.346956 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.347585 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.347711 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.352080 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:29.352091 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:29.352104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:29.352136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:29.355163 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:29.355835 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:29.355851 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:29.355854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:29.355966 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:29.356344 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:29.356350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:29.356355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:29.356713 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:29.357011 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:29.357321 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 02:53:29.357530 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:53:29.358659 25901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:29.447778 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25778
I20260621 02:53:29.467027 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25911
Jun 21 02:53:29 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:53:29.484756 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:29.467818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.11.109.3:35051)
0621 02:53:29.468029 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:29.468036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:29.468069 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:53:29.468205 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:29.468211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:29.468301 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:29.468432 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:29.469870 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.471130 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.472393 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:29.473126 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:29.473290 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:29.477657 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:29.477678 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:29.477701 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:29.477745 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:29.481905 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:29.482524 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:29.482539 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:29.482543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:29.482723 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:29.483067 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:29.483072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:29.483075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:29.483575 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:29.483907 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:29.484074 (+ 167us) server_negotiation.cc:299] Negotiation successful
0621 02:53:29.484311 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 02:53:29.485677 26034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:29.606622 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25911
I20260621 02:53:29.622390 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 25551
I20260621 02:53:29.623575 25613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:29.730901 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 25551
I20260621 02:53:29.751688 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45657
--webserver_interface=127.11.109.62
--webserver_port=33883
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:30.070307 26088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:30.070557 26088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:30.070720 26088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:30.077590 26088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:30.077684 26088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:30.077781 26088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:30.077886 26088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:30.077950 26088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:30.087150 26088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=33883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:30.088317 26088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:30.089550 26088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:30.111822 26094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:30.112313 26097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:30.111704 26095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:30.112763 26088 server_base.cc:1034] running on GCE node
I20260621 02:53:30.113677 26088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:30.115176 26088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:30.116410 26088 hybrid_clock.cc:648] HybridClock initialized: now 1782010410116365 us; error 63 us; skew 500 ppm
I20260621 02:53:30.116816 26088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:30 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010410, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:30.120514 26088 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:30.122177 26088 webserver.cc:460] Webserver started at http://127.11.109.62:33883/ using document root <none> and password file <none>
I20260621 02:53:30.122886 26088 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:30.123051 26088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:30.128618 26088 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260621 02:53:30.131453 26104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:30.132743 26088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:53:30.132927 26088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "85386363d27e4eb0a652ab3dcdd640fa"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "d4e6834fdace63f7f4fb26a7e22969ea"
server_key_iv: "4ce88ed67851cd5016a00f0f6efb91c5"
server_key_version: "encryptionkey@0"
I20260621 02:53:30.134025 26088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:30.147449 26088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:30.151618 26088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:30.151872 26088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:30.170317 26088 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45657
I20260621 02:53:30.170395 26155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45657 every 8 connection(s)
I20260621 02:53:30.171854 26088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:30.178682 26156 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:53:30.181834 26156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap starting.
I20260621 02:53:30.182343 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26088
I20260621 02:53:30.183135 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:45117
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:45657
--webserver_port=37149
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:30.206147 26156 log.cc:824] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:30.221684 26156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:53:30.222333 26156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap complete.
I20260621 02:53:30.235170 26156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:30.236326 26156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85386363d27e4eb0a652ab3dcdd640fa, State: Initialized, Role: FOLLOWER
I20260621 02:53:30.237037 26156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:30.237437 26156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:30.237567 26156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:30.237761 26156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:53:30.241621 26156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:30.242134 26156 leader_election.cc:302] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85386363d27e4eb0a652ab3dcdd640fa; no voters:
I20260621 02:53:30.242695 26156 leader_election.cc:288] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:53:30.242883 26160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:53:30.244143 26160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 LEADER]: Becoming Leader. State: Replica: 85386363d27e4eb0a652ab3dcdd640fa, State: Running, Role: LEADER
I20260621 02:53:30.244788 26156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:30.244808 26160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:30.248096 26162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85386363d27e4eb0a652ab3dcdd640fa. Latest consensus state: current_term: 2 leader_uuid: "85386363d27e4eb0a652ab3dcdd640fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } } }
I20260621 02:53:30.248623 26162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:30.249794 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85386363d27e4eb0a652ab3dcdd640fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } } }
I20260621 02:53:30.253980 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:30.262027 26169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:30.268321 26169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:30.269657 26169 catalog_manager.cc:1259] Loaded cluster ID: 0ab291d5ec004a6aa56a02096465e66b
I20260621 02:53:30.269752 26169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:30.272590 26169 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 02:53:30.289985 26169 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 02:53:30.298578 26169 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:30.300165 26169 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Loaded TSK: 0
I20260621 02:53:30.301615 26169 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:53:30.540020 26158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:30.540282 26158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:30.540436 26158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:30.547727 26158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:30.547821 26158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:30.547981 26158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:53:30.557451 26158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:45117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=37149
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:30.558665 26158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:30.559976 26158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:30.574244 26184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:30.574268 26185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:30.575157 26187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:30.575423 26158 server_base.cc:1034] running on GCE node
I20260621 02:53:30.576047 26158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:30.577080 26158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:30.578292 26158 hybrid_clock.cc:648] HybridClock initialized: now 1782010410578238 us; error 45 us; skew 500 ppm
I20260621 02:53:30.578619 26158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:30 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010410, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:30.585595 26158 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:53:30.587235 26158 webserver.cc:460] Webserver started at http://127.11.109.1:37149/ using document root <none> and password file <none>
I20260621 02:53:30.587889 26158 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:30.588028 26158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:30.593384 26158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:53:30.596042 26194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:30.597519 26158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:53:30.597680 26158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f55824adbd5645c5a14f5afa81f9c7bc"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "272c8f4411dfc822424c98e422f76ca4"
server_key_iv: "11e66ec616893a8293700dea076388ec"
server_key_version: "encryptionkey@0"
I20260621 02:53:30.598580 26158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:30.617873 26158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:30.621712 26158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:30.621984 26158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:30.622903 26158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:30.624161 26158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:30.624274 26158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:30.624409 26158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:30.624511 26158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:30.660163 26158 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:45117
I20260621 02:53:30.660205 26306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:45117 every 8 connection(s)
I20260621 02:53:30.661718 26158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:53:30.668485 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26158
I20260621 02:53:30.669272 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:33029
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:45657
--webserver_port=34007
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:53:30 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010410, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:30.686069 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:30.664441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:50797 (local address 127.11.109.62:45657)
0621 02:53:30.665005 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:30.665028 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:30.665665 (+ 637us) server_negotiation.cc:407] Connection header received
0621 02:53:30.667787 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:30.667799 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:30.668280 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:30.668703 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:30.670360 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:30.672294 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:30.673893 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:30.675006 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:30.675201 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:30.678828 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:30.678835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:30.678849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:30.678884 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:30.681800 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:30.682726 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:30.682731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:30.682736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:30.682903 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:30.683402 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:30.683405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:30.683407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:30.683806 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:30.683962 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:30.684844 (+ 882us) server_negotiation.cc:299] Negotiation successful
0621 02:53:30.685140 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
I20260621 02:53:30.686600 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:30.664653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:50797)
0621 02:53:30.665504 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:30.665548 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:30.667311 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:30.668908 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:30.668915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:30.669585 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:30.670121 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:30.670137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:30.672465 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:30.672469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:30.673739 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:30.673752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:30.675252 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:30.675258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:30.675447 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:30.676143 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:30.676169 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:30.678575 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:30.681936 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:30.681947 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:30.681963 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:30.682599 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:30.683034 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:30.683041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:30.683048 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:30.683272 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:30.683944 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:30.683954 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:30.684372 (+ 418us) client_negotiation.cc:769] Sending connection context
0621 02:53:30.684666 (+ 294us) client_negotiation.cc:240] Negotiation successful
0621 02:53:30.685258 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":378,"threads_started":1}
I20260621 02:53:30.689316 26307 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45657
I20260621 02:53:30.689661 26307 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:30.690599 26307 heartbeater.cc:511] Master 127.11.109.62:45657 requested a full tablet report, sending...
I20260621 02:53:30.692770 26121 ts_manager.cc:194] Registered new tserver with Master: f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117)
I20260621 02:53:30.694260 26121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:50797
W20260621 02:53:30.985033 26311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:30.985387 26311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:30.985556 26311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:30.992754 26311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:30.992874 26311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:30.993057 26311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:53:31.002527 26311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:33029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=34007
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:31.003767 26311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:31.004987 26311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:31.017877 26319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:31.017946 26317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:31.017961 26316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:31.018834 26311 server_base.cc:1034] running on GCE node
I20260621 02:53:31.019379 26311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:31.020202 26311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:31.021420 26311 hybrid_clock.cc:648] HybridClock initialized: now 1782010411021386 us; error 47 us; skew 500 ppm
I20260621 02:53:31.021720 26311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:31 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010411, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:31.024747 26311 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:53:31.026095 26311 webserver.cc:460] Webserver started at http://127.11.109.2:34007/ using document root <none> and password file <none>
I20260621 02:53:31.026728 26311 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:31.026880 26311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:31.031877 26311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:53:31.034399 26326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:31.035594 26311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:53:31.035770 26311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "44bc2988640b4c719d5b99cfa9601bcc"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "2f390a2774f3926bbee2c1ec69ed5dbc"
server_key_iv: "9898763847801ea630e4543b4e786889"
server_key_version: "encryptionkey@0"
I20260621 02:53:31.036691 26311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:31.083935 26311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:31.088207 26311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:31.088490 26311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:31.089490 26311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:31.090855 26311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:31.090988 26311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:31.091116 26311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:31.091219 26311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:31.124369 26311 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:33029
I20260621 02:53:31.124456 26438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:33029 every 8 connection(s)
I20260621 02:53:31.125778 26311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:53:31.133483 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26311
I20260621 02:53:31.134071 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:35051
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:45657
--webserver_port=41081
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:53:31 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010411, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:31.145961 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.128231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:56397 (local address 127.11.109.62:45657)
0621 02:53:31.128440 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.128444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.129787 (+ 1343us) server_negotiation.cc:407] Connection header received
0621 02:53:31.131635 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.131652 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.131735 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.131864 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:31.133425 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.134201 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.135729 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.136169 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.136312 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.139821 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:31.139827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:31.139832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:31.139914 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:31.142707 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:31.143643 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:31.143647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:31.143649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:31.143766 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:31.144238 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:31.144241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:31.144244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:31.144495 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:31.144610 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.145490 (+ 880us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.145663 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 02:53:31.147289 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.128737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:56397)
0621 02:53:31.129601 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.129647 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.131362 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.132095 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.132102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.132562 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:31.133196 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.133248 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.134403 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.134407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.135580 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.135591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.136414 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.136421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.136636 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.137342 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:31.137399 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:31.139633 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:31.142845 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:31.142856 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:31.142875 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:31.143523 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:31.143870 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:31.143877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:31.143884 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:31.144127 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:31.144614 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:31.144625 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:31.145045 (+ 420us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.145348 (+ 303us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.145880 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":256,"threads_started":1}
I20260621 02:53:31.148694 26439 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45657
I20260621 02:53:31.148999 26439 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:31.149745 26439 heartbeater.cc:511] Master 127.11.109.62:45657 requested a full tablet report, sending...
I20260621 02:53:31.151082 26121 ts_manager.cc:194] Registered new tserver with Master: 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
I20260621 02:53:31.151868 26121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:56397
W20260621 02:53:31.447623 26442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:31.447907 26442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:31.448073 26442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:31.455063 26442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:31.455199 26442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:31.455381 26442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:53:31.464771 26442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:35051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=41081
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:31.466017 26442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:31.467303 26442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:31.479723 26447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:31.480131 26448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:31.480140 26450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:31.480655 26442 server_base.cc:1034] running on GCE node
I20260621 02:53:31.481339 26442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:31.482208 26442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:31.483460 26442 hybrid_clock.cc:648] HybridClock initialized: now 1782010411483397 us; error 52 us; skew 500 ppm
I20260621 02:53:31.483776 26442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:31 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010411, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:31.486822 26442 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:53:31.488183 26442 webserver.cc:460] Webserver started at http://127.11.109.3:41081/ using document root <none> and password file <none>
I20260621 02:53:31.488819 26442 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:31.488976 26442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:31.494393 26442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:53:31.496827 26457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:31.498260 26442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:53:31.498416 26442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "a457d954efb14a6c937bdeff7d4536d3"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "be0a1366f9de2fb9004d016a7c81135f"
server_key_iv: "ca2b1aa0511ca1a28ea96d34829e37d0"
server_key_version: "encryptionkey@0"
I20260621 02:53:31.499326 26442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:31.521126 26442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:31.525302 26442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:31.525568 26442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:31.526461 26442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:31.527729 26442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:31.527861 26442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:31.527993 26442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:31.528084 26442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:31.561005 26442 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35051
I20260621 02:53:31.561097 26569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35051 every 8 connection(s)
I20260621 02:53:31.562541 26442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:53:31.566687 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26442
Jun 21 02:53:31 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010411, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:31.582492 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.567618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.11.109.62:45657)
0621 02:53:31.568722 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.568728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.568745 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:53:31.568809 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.568817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.568892 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.569013 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:31.570685 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.571531 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.572862 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.573466 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.573564 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.576490 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:31.576498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:31.576503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:31.576540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:31.579492 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:31.580284 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:31.580288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:31.580290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:31.580458 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:31.580880 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:31.580885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:31.580888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:31.581268 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:31.581513 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.581969 (+ 456us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.582199 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":883,"threads_started":1}
I20260621 02:53:31.583757 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.565066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:37551 (local address 127.11.109.62:45657)
0621 02:53:31.565305 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.565311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.566520 (+ 1209us) server_negotiation.cc:407] Connection header received
0621 02:53:31.569077 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.569082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.569169 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.569320 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:31.570735 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.571534 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.573154 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.573773 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.573894 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.577282 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:31.577289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:31.577295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:31.577349 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:31.580455 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:31.581370 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:31.581375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:31.581378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:31.581522 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:31.582140 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:31.582148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:31.582151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:31.582455 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:31.582560 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.583339 (+ 779us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.583530 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":1792}
I20260621 02:53:31.584772 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.565610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:37551)
0621 02:53:31.566338 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.566391 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.568787 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.569508 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.569515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.569984 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:31.570527 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.570542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.571691 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.571701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.573018 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.573031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.573983 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.573988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.574178 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.574702 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:31.574725 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:31.577012 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:31.580587 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:31.580595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:31.580612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:31.581235 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:31.581664 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:31.581667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:31.581669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:31.582008 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:31.582611 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:31.582619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:31.582976 (+ 357us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.583236 (+ 260us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.583810 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":272,"threads_started":1}
I20260621 02:53:31.586078 26570 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45657
I20260621 02:53:31.586395 26570 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:31.587075 26570 heartbeater.cc:511] Master 127.11.109.62:45657 requested a full tablet report, sending...
I20260621 02:53:31.588395 26121 ts_manager.cc:194] Registered new tserver with Master: a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051)
I20260621 02:53:31.589252 26121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:37551
I20260621 02:53:31.597505 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:53:31.603941 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.598416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.11.109.62:45657)
0621 02:53:31.598567 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.598572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.599030 (+ 458us) server_negotiation.cc:407] Connection header received
0621 02:53:31.599184 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.599187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.599307 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.599418 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:53:31.599756 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.600445 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.601897 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.602291 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.602377 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.602960 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:53:31.603436 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:53:31.603536 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.603644 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.603742 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:53:31.604698 26121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:53:31.612007 26121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:53:31.646740 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.623654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.11.109.1:45117)
0621 02:53:31.635876 (+ 12222us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.635889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.636780 (+ 891us) server_negotiation.cc:407] Connection header received
0621 02:53:31.636912 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.636923 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.637211 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.637420 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.638508 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.640503 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.644003 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.645494 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.645628 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.645804 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.646272 (+ 468us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.646440 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11998,"thread_start_us":12222,"threads_started":1}
I20260621 02:53:31.646891 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.623007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.11.109.3:35051)
0621 02:53:31.636413 (+ 13406us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.636419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.636445 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:53:31.636516 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.636521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.636802 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.636994 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.638343 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.640642 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.644136 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.645778 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.645895 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.646103 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.646392 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.646549 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13269,"thread_start_us":15033,"threads_started":1}
I20260621 02:53:31.647203 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.623095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36750)
0621 02:53:31.636529 (+ 13434us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.636547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.636653 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.637702 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.637710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.637743 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.638256 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.638271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.640715 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.640723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.643816 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.643830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.645696 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.645704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.645928 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.645986 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.646220 (+ 234us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.646437 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13295,"mutex_wait_us":52,"thread_start_us":10749,"threads_started":1}
I20260621 02:53:31.648113 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.622840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35051 (local address 127.0.0.1:58148)
0621 02:53:31.634229 (+ 11389us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.634262 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.634425 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.637319 (+ 2894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.637326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.637374 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.637869 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.637892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.640877 (+ 2985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.640887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.643957 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.643971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.645974 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.645980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.646141 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.646173 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.646244 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.647169 (+ 925us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11143,"mutex_wait_us":811,"thread_start_us":13432,"threads_started":1}
I20260621 02:53:31.651443 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.636641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:33029 (local address 127.0.0.1:51860)
0621 02:53:31.643168 (+ 6527us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.643185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.643294 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.644492 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.644497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.644524 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.644912 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.644922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.646787 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.646792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.649563 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.649575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.650804 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.650810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.650982 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.651014 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.651106 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.651225 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6393,"thread_start_us":6762,"threads_started":1}
I20260621 02:53:31.651207 26242 tablet_service.cc:1467] Processing CreateTablet for tablet 11b2d4b9c7a449dbbb5848b77926df4b (DEFAULT_TABLE table=test-table [id=1faff1f4eacb465cb248ef500331c7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:31.652205 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.637036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.11.109.2:33029)
0621 02:53:31.643890 (+ 6854us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.643901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.643941 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:53:31.644036 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.644044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.644241 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.644472 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.645088 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.646602 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.649699 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.650584 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.650738 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.650931 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.651821 (+ 890us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.652000 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6685,"thread_start_us":6856,"threads_started":1}
I20260621 02:53:31.651688 26505 tablet_service.cc:1467] Processing CreateTablet for tablet 11b2d4b9c7a449dbbb5848b77926df4b (DEFAULT_TABLE table=test-table [id=1faff1f4eacb465cb248ef500331c7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:31.652791 26242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b2d4b9c7a449dbbb5848b77926df4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:31.653000 26505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b2d4b9c7a449dbbb5848b77926df4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:31.654932 26374 tablet_service.cc:1467] Processing CreateTablet for tablet 11b2d4b9c7a449dbbb5848b77926df4b (DEFAULT_TABLE table=test-table [id=1faff1f4eacb465cb248ef500331c7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:31.655884 26374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b2d4b9c7a449dbbb5848b77926df4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:31.673404 26588 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Bootstrap starting.
I20260621 02:53:31.674825 26586 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Bootstrap starting.
I20260621 02:53:31.676441 26588 tablet_bootstrap.cc:654] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:31.677975 26588 log.cc:824] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:31.678471 26587 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Bootstrap starting.
I20260621 02:53:31.678944 26586 tablet_bootstrap.cc:654] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:31.680087 26586 log.cc:824] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:31.682296 26587 tablet_bootstrap.cc:654] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:31.683022 26588 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: No bootstrap required, opened a new log
I20260621 02:53:31.683228 26586 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: No bootstrap required, opened a new log
I20260621 02:53:31.683320 26588 ts_tablet_manager.cc:1397] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:53:31.683508 26586 ts_tablet_manager.cc:1397] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:53:31.683493 26587 log.cc:824] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:31.686357 26587 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: No bootstrap required, opened a new log
I20260621 02:53:31.686601 26587 ts_tablet_manager.cc:1397] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:53:31.691056 26588 raft_consensus.cc:348] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.691401 26588 raft_consensus.cc:374] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:31.691531 26588 raft_consensus.cc:729] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44bc2988640b4c719d5b99cfa9601bcc, State: Initialized, Role: FOLLOWER
I20260621 02:53:31.692123 26588 consensus_queue.cc:260] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.693138 26439 heartbeater.cc:503] Master 127.11.109.62:45657 was elected leader, sending a full tablet report...
I20260621 02:53:31.693818 26588 ts_tablet_manager.cc:1428] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260621 02:53:31.694051 26586 raft_consensus.cc:348] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.694430 26586 raft_consensus.cc:374] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:31.694535 26586 raft_consensus.cc:729] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55824adbd5645c5a14f5afa81f9c7bc, State: Initialized, Role: FOLLOWER
I20260621 02:53:31.695088 26586 consensus_queue.cc:260] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.696136 26307 heartbeater.cc:503] Master 127.11.109.62:45657 was elected leader, sending a full tablet report...
I20260621 02:53:31.696681 26586 ts_tablet_manager.cc:1428] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260621 02:53:31.696988 26587 raft_consensus.cc:348] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.697410 26587 raft_consensus.cc:374] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:31.697520 26587 raft_consensus.cc:729] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a457d954efb14a6c937bdeff7d4536d3, State: Initialized, Role: FOLLOWER
I20260621 02:53:31.698026 26587 consensus_queue.cc:260] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.700698 26570 heartbeater.cc:503] Master 127.11.109.62:45657 was elected leader, sending a full tablet report...
I20260621 02:53:31.701284 26587 ts_tablet_manager.cc:1428] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260621 02:53:31.814174 26571 tablet.cc:2369] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:53:31.877743 26440 tablet.cc:2369] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:31.881847 26593 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:31.882175 26593 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.883996 26593 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
I20260621 02:53:31.890604 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.884376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:34689 (local address 127.11.109.3:35051)
0621 02:53:31.884601 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.884609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.885199 (+ 590us) server_negotiation.cc:407] Connection header received
0621 02:53:31.885566 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.885571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.885657 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.885709 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.886398 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.887109 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.888879 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.889713 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.889894 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.889981 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.890232 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.890355 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:53:31.890698 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.884337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35051 (local address 127.11.109.1:34689)
0621 02:53:31.885084 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.885103 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.885316 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.885944 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.885948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.885969 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.886262 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.886269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.887242 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.887247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.888737 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.888752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.889907 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.889913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.890077 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.890100 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.890209 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.890338 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":411,"threads_started":1}
I20260621 02:53:31.890956 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.885452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:57703 (local address 127.11.109.2:33029)
0621 02:53:31.885661 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.885667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.885901 (+ 234us) server_negotiation.cc:407] Connection header received
0621 02:53:31.886103 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.886108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.886169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.886257 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.886782 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.887429 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.889149 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.890029 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.890221 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.890290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.890561 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.890679 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:53:31.891232 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.885203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:33029 (local address 127.11.109.1:57703)
0621 02:53:31.885792 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.885812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.885958 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.886290 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.886294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.886319 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.886635 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.886643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.887573 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.887579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.888991 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.889003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.890234 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.890240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.890406 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.890429 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.890539 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.890658 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":1,"thread_start_us":283,"threads_started":1}
I20260621 02:53:31.891791 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b2d4b9c7a449dbbb5848b77926df4b" candidate_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a457d954efb14a6c937bdeff7d4536d3" is_pre_election: true
I20260621 02:53:31.892271 26394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b2d4b9c7a449dbbb5848b77926df4b" candidate_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bc2988640b4c719d5b99cfa9601bcc" is_pre_election: true
I20260621 02:53:31.892387 26525 raft_consensus.cc:2454] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55824adbd5645c5a14f5afa81f9c7bc in term 0.
I20260621 02:53:31.892802 26394 raft_consensus.cc:2454] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55824adbd5645c5a14f5afa81f9c7bc in term 0.
I20260621 02:53:31.893070 26592 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:31.893126 26198 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a457d954efb14a6c937bdeff7d4536d3, f55824adbd5645c5a14f5afa81f9c7bc; no voters:
I20260621 02:53:31.893332 26592 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.893543 26593 raft_consensus.cc:2781] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:53:31.893683 26593 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:53:31.893797 26593 raft_consensus.cc:3037] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:31.894870 26592 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117)
I20260621 02:53:31.896552 26593 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.897375 26593 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 1 election: Requested vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
I20260621 02:53:31.897711 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b2d4b9c7a449dbbb5848b77926df4b" candidate_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a457d954efb14a6c937bdeff7d4536d3"
I20260621 02:53:31.897828 26394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b2d4b9c7a449dbbb5848b77926df4b" candidate_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44bc2988640b4c719d5b99cfa9601bcc"
I20260621 02:53:31.897928 26525 raft_consensus.cc:3037] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:31.898038 26394 raft_consensus.cc:3037] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:31.901407 26525 raft_consensus.cc:2454] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55824adbd5645c5a14f5afa81f9c7bc in term 1.
I20260621 02:53:31.901428 26394 raft_consensus.cc:2454] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55824adbd5645c5a14f5afa81f9c7bc in term 1.
I20260621 02:53:31.901854 26198 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44bc2988640b4c719d5b99cfa9601bcc, f55824adbd5645c5a14f5afa81f9c7bc; no voters:
I20260621 02:53:31.902472 26593 raft_consensus.cc:2781] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:31.904021 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.896552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:51543 (local address 127.11.109.1:45117)
0621 02:53:31.896769 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.896777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.896796 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:53:31.896847 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.896852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.896924 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.897016 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.897679 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.898943 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.901797 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.902875 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.902977 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.903063 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.903598 (+ 535us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.903728 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:53:31.904422 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.895241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35051 (local address 127.11.109.2:36657)
0621 02:53:31.896005 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.896026 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.896145 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.897600 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.897604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.897633 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.898050 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.898061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.899517 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.899522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.901817 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.901828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.903639 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.903645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.903793 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.903825 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.903952 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.904090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":387,"threads_started":1}
I20260621 02:53:31.904562 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.896789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:36657 (local address 127.11.109.3:35051)
0621 02:53:31.896961 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.896964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.896998 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:53:31.897052 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.897056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.897121 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.897241 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:31.898214 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.899331 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.901966 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.903366 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.903581 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.903685 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.903988 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.904179 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:53:31.905110 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.895252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.11.109.2:51543)
0621 02:53:31.896415 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:31.896433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:31.896545 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:31.897051 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:31.897056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:31.897082 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:31.897432 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.897498 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.899116 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.899120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.901618 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:31.901633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.903272 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.903276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:31.903403 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.903424 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:53:31.904354 (+ 930us) client_negotiation.cc:240] Negotiation successful
0621 02:53:31.904824 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":25,"thread_start_us":852,"threads_started":1}
I20260621 02:53:31.905247 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b2d4b9c7a449dbbb5848b77926df4b" candidate_uuid: "44bc2988640b4c719d5b99cfa9601bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a457d954efb14a6c937bdeff7d4536d3" is_pre_election: true
I20260621 02:53:31.905576 26525 raft_consensus.cc:2379] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44bc2988640b4c719d5b99cfa9601bcc in current term 1: Already voted for candidate f55824adbd5645c5a14f5afa81f9c7bc in this term.
I20260621 02:53:31.906317 26262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b2d4b9c7a449dbbb5848b77926df4b" candidate_uuid: "44bc2988640b4c719d5b99cfa9601bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" is_pre_election: true
I20260621 02:53:31.909407 26593 raft_consensus.cc:686] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 LEADER]: Becoming Leader. State: Replica: f55824adbd5645c5a14f5afa81f9c7bc, State: Running, Role: LEADER
I20260621 02:53:31.909924 26593 consensus_queue.cc:237] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:31.913767 26329 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44bc2988640b4c719d5b99cfa9601bcc; no voters: a457d954efb14a6c937bdeff7d4536d3, f55824adbd5645c5a14f5afa81f9c7bc
W20260621 02:53:31.913955 26308 tablet.cc:2369] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:31.914180 26592 raft_consensus.cc:2726] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 02:53:31.915136 26120 catalog_manager.cc:5495] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1). New cstate: current_term: 1 leader_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } health_report { overall_health: UNKNOWN } } }
I20260621 02:53:31.954679 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.949142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.11.109.1:45117)
0621 02:53:31.949336 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.949342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.949480 (+ 138us) server_negotiation.cc:407] Connection header received
0621 02:53:31.949713 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.949717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.949800 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.949958 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:53:31.950321 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.951070 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.952527 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.952945 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:31.953073 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:31.953609 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:53:31.954097 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:53:31.954258 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:31.954317 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 02:53:31.954393 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:53:31.962466 26525 raft_consensus.cc:1261] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Refusing update from remote peer f55824adbd5645c5a14f5afa81f9c7bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:31.962495 26394 raft_consensus.cc:1261] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Refusing update from remote peer f55824adbd5645c5a14f5afa81f9c7bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:31.963460 26599 consensus_queue.cc:1035] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:31.963888 26593 consensus_queue.cc:1035] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:31.986632 26120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:31.987591 26120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:31.991429 26120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Sending DeleteTablet for 3 replicas of tablet 11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:31.992630 26242 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b2d4b9c7a449dbbb5848b77926df4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:31 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:36750
I20260621 02:53:31.992671 26374 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b2d4b9c7a449dbbb5848b77926df4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:31 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:51860
I20260621 02:53:31.993026 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26158
I20260621 02:53:31.993654 26505 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b2d4b9c7a449dbbb5848b77926df4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:31 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:58148
I20260621 02:53:32.002784 26613 tablet_replica.cc:330] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: stopping tablet replica
I20260621 02:53:32.003468 26613 raft_consensus.cc:2229] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:32.004467 26612 tablet_replica.cc:330] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: stopping tablet replica
I20260621 02:53:32.004814 26612 raft_consensus.cc:2229] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:32.005265 26612 raft_consensus.cc:2258] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:32.005334 26613 raft_consensus.cc:2258] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:32.007520 26613 ts_tablet_manager.cc:1905] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:32.007520 26612 ts_tablet_manager.cc:1905] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:32.014472 26612 ts_tablet_manager.cc:1918] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:32.014473 26613 ts_tablet_manager.cc:1918] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:32.014633 26612 log.cc:1196] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:32.014794 26613 log.cc:1196] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:32.015215 26612 ts_tablet_manager.cc:1939] T 11b2d4b9c7a449dbbb5848b77926df4b P 44bc2988640b4c719d5b99cfa9601bcc: Deleting consensus metadata
I20260621 02:53:32.015456 26613 ts_tablet_manager.cc:1939] T 11b2d4b9c7a449dbbb5848b77926df4b P a457d954efb14a6c937bdeff7d4536d3: Deleting consensus metadata
I20260621 02:53:32.017160 26108 catalog_manager.cc:4841] TS a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): tablet 11b2d4b9c7a449dbbb5848b77926df4b (table test-table [id=1faff1f4eacb465cb248ef500331c7b8]) successfully deleted
I20260621 02:53:32.017544 26108 catalog_manager.cc:4841] TS 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): tablet 11b2d4b9c7a449dbbb5848b77926df4b (table test-table [id=1faff1f4eacb465cb248ef500331c7b8]) successfully deleted
I20260621 02:53:32.018009 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:31.993970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.11.109.1:45117)
0621 02:53:31.994152 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:31.994160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:31.994305 (+ 145us) server_negotiation.cc:407] Connection header received
0621 02:53:31.994550 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:31.994558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:31.994657 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:31.994778 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:31.996965 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:31.998391 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:32.000925 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:32.001919 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:32.002144 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:32.007324 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:32.007344 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:32.007367 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:32.007428 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:32.012666 (+ 5238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:32.014409 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:32.014423 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:32.014429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:32.014638 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:32.015208 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:32.015216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:32.015221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:32.015841 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:32.016194 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:32.017434 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0621 02:53:32.017673 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:53:32.018741 26302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:32.113487 26611 tablet_replica.cc:330] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: stopping tablet replica
I20260621 02:53:32.113855 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26158
W20260621 02:53:32.131857 26107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.109.1:45117 (error 108)
I20260621 02:53:32.132043 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26311
I20260621 02:53:32.135808 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.135356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36758)
0621 02:53:32.135537 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.135656 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 02:53:32.136021 26107 catalog_manager.cc:4584] TS f55824adbd5645c5a14f5afa81f9c7bc (127.11.109.1:45117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11b2d4b9c7a449dbbb5848b77926df4b: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
I20260621 02:53:32.146564 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.132975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.11.109.2:33029)
0621 02:53:32.133145 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:32.133151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:32.133167 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:53:32.133273 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:32.133281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:32.133369 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:32.133470 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:32.134786 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:32.135901 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:32.137460 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:32.137938 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:32.138074 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:32.140830 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:32.140839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:32.140851 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:32.140927 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:32.143724 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:32.144453 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:32.144469 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:32.144471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:32.144595 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:32.144976 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:32.144983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:32.144986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:32.145424 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:32.145711 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:32.146023 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 02:53:32.146229 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:53:32.147141 26434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:32.189685 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.189256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36760)
0621 02:53:32.189446 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.189545 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 02:53:32.190833 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.190519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36776)
0621 02:53:32.190643 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.190716 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 02:53:32.251485 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.250996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36782)
0621 02:53:32.251178 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.251281 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 02:53:32.251847 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26311
I20260621 02:53:32.257851 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.257418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36798)
0621 02:53:32.257604 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.257699 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:53:32.268797 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26442
I20260621 02:53:32.282793 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.269559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.11.109.3:35051)
0621 02:53:32.269742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:32.269747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:32.269766 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:53:32.269966 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:32.269975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:32.270039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:32.270137 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:32.271464 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:32.272167 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:32.273684 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:32.274107 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:32.274230 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:32.276857 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:32.276884 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:32.276900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:32.276936 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:32.280021 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:32.280688 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:32.280715 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:32.280725 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:32.280884 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:32.281309 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:32.281316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:32.281319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:32.281658 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:32.281984 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:32.282283 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 02:53:32.282481 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:53:32.283449 26565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:32.338297 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.337760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36812)
0621 02:53:32.338027 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.338112 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260621 02:53:32.339365 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:32.338999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45117 (local address 127.0.0.1:36828)
0621 02:53:32.339190 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:32.339260 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 02:53:32.371572 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26442
I20260621 02:53:32.388196 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26088
I20260621 02:53:32.389358 26151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:32.500932 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26088
I20260621 02:53:32.522809 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45657
--webserver_interface=127.11.109.62
--webserver_port=33883
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:32.839020 26619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:32.839287 26619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:32.839376 26619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:32.846290 26619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:32.846379 26619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:32.846426 26619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:32.846478 26619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:32.846517 26619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:32.855603 26619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=33883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:32.856714 26619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:32.857896 26619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:32.869112 26624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:32.869081 26625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:32.869228 26627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:32.869874 26619 server_base.cc:1034] running on GCE node
I20260621 02:53:32.870596 26619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:32.871945 26619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:32.873257 26619 hybrid_clock.cc:648] HybridClock initialized: now 1782010412873145 us; error 45 us; skew 500 ppm
I20260621 02:53:32.873625 26619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:32 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010412, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:32.876583 26619 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:32.878065 26619 webserver.cc:460] Webserver started at http://127.11.109.62:33883/ using document root <none> and password file <none>
I20260621 02:53:32.878743 26619 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:32.878887 26619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:32.884125 26619 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 02:53:32.886916 26634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:32.888216 26619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:53:32.888410 26619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "85386363d27e4eb0a652ab3dcdd640fa"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "d4e6834fdace63f7f4fb26a7e22969ea"
server_key_iv: "4ce88ed67851cd5016a00f0f6efb91c5"
server_key_version: "encryptionkey@0"
I20260621 02:53:32.889405 26619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:32.907490 26619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:32.911345 26619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:32.911607 26619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:32.929947 26619 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45657
I20260621 02:53:32.929994 26685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45657 every 8 connection(s)
I20260621 02:53:32.931382 26619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:32.933323 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26619
I20260621 02:53:32.934111 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:45117
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:45657
--webserver_port=37149
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:32.938095 26686 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:53:32.941258 26686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap starting.
I20260621 02:53:32.960913 26686 log.cc:824] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:32.977835 26686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:53:32.978386 26686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa: Bootstrap complete.
I20260621 02:53:32.987475 26686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:32.988356 26686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85386363d27e4eb0a652ab3dcdd640fa, State: Initialized, Role: FOLLOWER
I20260621 02:53:32.988986 26686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:32.989248 26686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:32.989379 26686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:32.989574 26686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:53:32.993701 26686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:32.994264 26686 leader_election.cc:302] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85386363d27e4eb0a652ab3dcdd640fa; no voters:
I20260621 02:53:32.994895 26686 leader_election.cc:288] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 02:53:32.995064 26690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:53:32.996112 26690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [term 3 LEADER]: Becoming Leader. State: Replica: 85386363d27e4eb0a652ab3dcdd640fa, State: Running, Role: LEADER
I20260621 02:53:32.996654 26690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } }
I20260621 02:53:32.996909 26686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:32.999058 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85386363d27e4eb0a652ab3dcdd640fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } } }
I20260621 02:53:32.999405 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:33.001103 26692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85386363d27e4eb0a652ab3dcdd640fa. Latest consensus state: current_term: 3 leader_uuid: "85386363d27e4eb0a652ab3dcdd640fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85386363d27e4eb0a652ab3dcdd640fa" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45657 } } }
I20260621 02:53:33.001391 26692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85386363d27e4eb0a652ab3dcdd640fa [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:33.001849 26695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:33.011914 26695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:33.013873 26695 catalog_manager.cc:1259] Loaded cluster ID: 0ab291d5ec004a6aa56a02096465e66b
I20260621 02:53:33.014050 26695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 02:53:33.016656 26695 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa41868e9ad google::LogMessage::Fail() at ??:0
@ 0x7fa41869188a google::LogMessage::SendToLog() at ??:0
@ 0x7fa41868e4de google::LogMessage::Flush() at ??:0
@ 0x7fa41868fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa42e0c1512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa42e0be976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260621 02:53:33.044726 26709 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fa42e146610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa42e146439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa41b2da345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa41b2dd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa41b2dd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa41b2b8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa41e5926db start_thread at ??:0
@ 0x7fa41752861f clone at ??:0
W20260621 02:53:33.299579 26688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:33.299881 26688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:33.300050 26688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:33.307325 26688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:33.307473 26688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:33.307639 26688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:53:33.317081 26688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:45117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=37149
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:33.318359 26688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:33.319625 26688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:33.332795 26728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:33.332891 26727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:33.332921 26730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:33.333467 26688 server_base.cc:1034] running on GCE node
I20260621 02:53:33.333984 26688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:33.334808 26688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:33.336009 26688 hybrid_clock.cc:648] HybridClock initialized: now 1782010413335966 us; error 36 us; skew 500 ppm
I20260621 02:53:33.336254 26688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:33 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010413, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:33.339272 26688 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:53:33.340696 26688 webserver.cc:460] Webserver started at http://127.11.109.1:37149/ using document root <none> and password file <none>
I20260621 02:53:33.341428 26688 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:33.341581 26688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:33.346652 26688 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:53:33.349483 26737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:33.350718 26688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:53:33.350891 26688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f55824adbd5645c5a14f5afa81f9c7bc"
format_stamp: "Formatted at 2026-06-21 02:53:27 on dist-test-slave-bc52"
server_key: "272c8f4411dfc822424c98e422f76ca4"
server_key_iv: "11e66ec616893a8293700dea076388ec"
server_key_version: "encryptionkey@0"
I20260621 02:53:33.351890 26688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:33.369074 26688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:33.372711 26688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:33.372987 26688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:33.373919 26688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:33.375608 26744 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260621 02:53:33.380818 26688 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 02:53:33.380978 26688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260621 02:53:33.381131 26688 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 02:53:33.384152 26688 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 02:53:33.384254 26688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260621 02:53:33.384589 26744 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Bootstrap starting.
I20260621 02:53:33.412132 26744 log.cc:824] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:33.423511 26688 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:45117
I20260621 02:53:33.423544 26851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:45117 every 8 connection(s)
I20260621 02:53:33.424711 26688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:53:33.426618 26744 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:53:33.426940 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26688
I20260621 02:53:33.427263 26744 tablet_bootstrap.cc:492] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Bootstrap complete.
I20260621 02:53:33.427899 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:33029
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:45657
--webserver_port=34007
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:53:33.429158 26744 ts_tablet_manager.cc:1397] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.017s
I20260621 02:53:33.439524 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.427743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:38267)
0621 02:53:33.437992 (+ 10249us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.438305 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9139,"thread_start_us":9347,"threads_started":1}
W20260621 02:53:33.440447 26852 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:45657 (0 consecutive failures): Network error: Failed to ping master at 127.11.109.62:45657: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
I20260621 02:53:33.441511 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.441031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:56245)
0621 02:53:33.441278 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.441363 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260621 02:53:33.441217 26744 raft_consensus.cc:348] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:33.442517 26744 raft_consensus.cc:729] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55824adbd5645c5a14f5afa81f9c7bc, State: Initialized, Role: FOLLOWER
I20260621 02:53:33.443068 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.442631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:59927)
0621 02:53:33.442817 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.442912 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260621 02:53:33.443365 26852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:53:33.443238 26744 consensus_queue.cc:260] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:33.444747 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.444433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:34741)
0621 02:53:33.444572 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.444646 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:53:33.444913 26744 ts_tablet_manager.cc:1428] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260621 02:53:33.445787 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.445484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:39013)
0621 02:53:33.445615 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.445686 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:53:33.753299 26855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:33.753572 26855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:33.753724 26855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:33.760970 26855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:33.761070 26855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:33.761270 26855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:53:33.770864 26855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:33029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=34007
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:33.772018 26855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:33.773350 26855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:33.787846 26861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:33.787784 26862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:33.787814 26864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:33.788479 26855 server_base.cc:1034] running on GCE node
I20260621 02:53:33.789058 26855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:33.789932 26855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:33.791122 26855 hybrid_clock.cc:648] HybridClock initialized: now 1782010413791090 us; error 26 us; skew 500 ppm
I20260621 02:53:33.791518 26855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:33 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010413, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:33.794548 26855 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:53:33.795912 26855 webserver.cc:460] Webserver started at http://127.11.109.2:34007/ using document root <none> and password file <none>
I20260621 02:53:33.796634 26855 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:33.796763 26855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:33.801784 26855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:53:33.804369 26871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:33.805686 26855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:53:33.805855 26855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "44bc2988640b4c719d5b99cfa9601bcc"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "2f390a2774f3926bbee2c1ec69ed5dbc"
server_key_iv: "9898763847801ea630e4543b4e786889"
server_key_version: "encryptionkey@0"
I20260621 02:53:33.806900 26855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:33.821961 26855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:33.825753 26855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:33.826013 26855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:33.826957 26855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:33.828224 26855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:33.828351 26855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:33.828505 26855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:33.828596 26855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:33.861951 26855 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:33029
I20260621 02:53:33.862032 26983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:33029 every 8 connection(s)
I20260621 02:53:33.863400 26855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:53:33.867770 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26855
I20260621 02:53:33.868419 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.866372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:44179)
0621 02:53:33.867051 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.867266 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260621 02:53:33.868513 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:35051
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:45657
--webserver_port=41081
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:33.869292 26984 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:45657 (0 consecutive failures): Network error: Failed to ping master at 127.11.109.62:45657: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
I20260621 02:53:33.870146 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.869830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:49217)
0621 02:53:33.869972 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.870048 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:53:33.871021 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:33.870716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:54059)
0621 02:53:33.870848 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:33.870915 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 02:53:33.871304 26984 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:53:34.181051 26987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:34.181354 26987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:34.181545 26987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:34.188580 26987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:34.188683 26987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:34.188874 26987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:53:34.198590 26987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:35051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=41081
--tserver_master_addrs=127.11.109.62:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:34.199784 26987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:34.201045 26987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:34.213663 26992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:34.213719 26995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:34.213742 26993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:34.215021 26987 server_base.cc:1034] running on GCE node
I20260621 02:53:34.215574 26987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:34.216442 26987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:34.217654 26987 hybrid_clock.cc:648] HybridClock initialized: now 1782010414217616 us; error 32 us; skew 500 ppm
I20260621 02:53:34.217953 26987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:34 dist-test-slave-bc52 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010414, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:34.220850 26987 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:53:34.222311 26987 webserver.cc:460] Webserver started at http://127.11.109.3:41081/ using document root <none> and password file <none>
I20260621 02:53:34.222951 26987 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:34.223093 26987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:34.227948 26987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 02:53:34.230511 27002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:34.231709 26987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:53:34.231873 26987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "a457d954efb14a6c937bdeff7d4536d3"
format_stamp: "Formatted at 2026-06-21 02:53:28 on dist-test-slave-bc52"
server_key: "be0a1366f9de2fb9004d016a7c81135f"
server_key_iv: "ca2b1aa0511ca1a28ea96d34829e37d0"
server_key_version: "encryptionkey@0"
I20260621 02:53:34.232765 26987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:34.245146 26987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:34.248553 26987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:34.248806 26987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:34.249697 26987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:34.250921 26987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:34.251035 26987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:34.251159 26987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:34.251246 26987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:34.283246 26987 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:35051
I20260621 02:53:34.283327 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:35051 every 8 connection(s)
I20260621 02:53:34.284677 26987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:53:34.289114 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 26987
I20260621 02:53:34.289981 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:34.287548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:41381)
0621 02:53:34.288354 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:34.288773 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":391,"threads_started":1}
W20260621 02:53:34.290819 27115 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:45657 (0 consecutive failures): Network error: Failed to ping master at 127.11.109.62:45657: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
I20260621 02:53:34.291576 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:34.291299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:55265)
0621 02:53:34.291425 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:34.291487 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:53:34.292338 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:34.292066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:47411)
0621 02:53:34.292175 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:34.292241 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 02:53:34.292572 27115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:53:34.447345 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:34.446577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:47979)
0621 02:53:34.447043 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:34.447154 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260621 02:53:34.872561 27120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:34.871751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:37343)
0621 02:53:34.872213 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:34.872335 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":194,"threads_started":1}
I20260621 02:53:35.293773 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.293014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:60217)
0621 02:53:35.293469 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:35.293570 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
I20260621 02:53:35.298328 27122 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:35.298601 27122 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:35.299981 27122 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
Jun 21 02:53:35 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010413, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
Jun 21 02:53:35 dist-test-slave-bc52 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010413, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:53:35.327514 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.300471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:50589 (local address 127.11.109.3:35051)
0621 02:53:35.302814 (+ 2343us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:35.302827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:35.302868 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:53:35.305123 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:35.305134 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:35.305601 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:35.306058 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:35.308741 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.310679 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.313658 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.314661 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.314841 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:35.319448 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:35.319460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:35.319478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:35.319534 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:35.323267 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:35.324531 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:35.324541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:35.324551 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:35.324739 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:35.325362 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:35.325369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:35.325372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:35.325879 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:35.326049 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:35.326736 (+ 687us) server_negotiation.cc:299] Negotiation successful
0621 02:53:35.327113 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2180,"thread_start_us":2033,"threads_started":1}
I20260621 02:53:35.327523 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.300222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:35051 (local address 127.11.109.1:50589)
0621 02:53:35.300723 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:35.300773 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:35.301427 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:35.306228 (+ 4801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:35.306241 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:35.307408 (+ 1167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:35.308520 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:35.308546 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.310884 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.310888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:35.313497 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:35.313511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.314830 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.314834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:35.315120 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:35.315932 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:35.315961 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:35.319124 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:35.323454 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:35.323463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:35.323478 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:35.324265 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:35.324868 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:35.324873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:35.324877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:35.325122 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:35.326024 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:35.326028 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:35.326257 (+ 229us) client_negotiation.cc:769] Sending connection context
0621 02:53:35.326459 (+ 202us) client_negotiation.cc:240] Negotiation successful
0621 02:53:35.327022 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":390,"spinlock_wait_cycles":21120,"thread_start_us":199,"threads_started":1}
I20260621 02:53:35.327535 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.301236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:40413 (local address 127.11.109.2:33029)
0621 02:53:35.302354 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:35.302373 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:35.302699 (+ 326us) server_negotiation.cc:407] Connection header received
0621 02:53:35.305347 (+ 2648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:35.305356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:35.305738 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:35.306057 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:35.308741 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.310679 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.313923 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.314863 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.315024 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:35.319594 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:35.319606 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:35.319620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:35.319672 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:35.323267 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:35.324422 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:35.324433 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:35.324443 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:35.324676 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:35.325243 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:35.325248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:35.325252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:35.325774 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:35.325903 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:35.326761 (+ 858us) server_negotiation.cc:299] Negotiation successful
0621 02:53:35.327143 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":671,"threads_started":1}
I20260621 02:53:35.327637 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.300814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:33029 (local address 127.11.109.1:40413)
0621 02:53:35.302569 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:35.302587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:35.303038 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:35.306228 (+ 3190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:35.306241 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:35.307408 (+ 1167us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:35.308520 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:35.308546 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.310879 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.310885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:35.313498 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:35.313511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:35.315034 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:35.315040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:35.315195 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:35.315932 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:35.315969 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:35.319338 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:35.323454 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:35.323463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:35.323478 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:35.324266 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:35.324820 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:35.324828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:35.324834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:35.325101 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:35.325921 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:35.325931 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:35.326257 (+ 326us) client_negotiation.cc:769] Sending connection context
0621 02:53:35.326495 (+ 238us) client_negotiation.cc:240] Negotiation successful
0621 02:53:35.327178 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1630,"mutex_wait_us":598,"spinlock_wait_cycles":27136,"thread_start_us":1458,"threads_started":1}
W20260621 02:53:35.330643 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
W20260621 02:53:35.330969 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:35.331113 26741 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f55824adbd5645c5a14f5afa81f9c7bc; no voters: 44bc2988640b4c719d5b99cfa9601bcc, a457d954efb14a6c937bdeff7d4536d3
I20260621 02:53:35.331525 27122 raft_consensus.cc:2726] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:53:35.448573 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.448141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:58259)
0621 02:53:35.448324 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:35.448430 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260621 02:53:35.874043 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:35.873456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:56051)
0621 02:53:35.873836 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:35.873907 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":213,"threads_started":1}
I20260621 02:53:36.295298 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:36.294560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:35803)
0621 02:53:36.295075 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:36.295151 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":314,"threads_started":1}
I20260621 02:53:36.450137 27129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:36.449386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:45615)
0621 02:53:36.449872 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:36.449950 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":213,"threads_started":1}
I20260621 02:53:36.875474 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:36.874816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:57635)
0621 02:53:36.875231 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:36.875314 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":228,"threads_started":1}
I20260621 02:53:37.297091 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:37.296180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:52791)
0621 02:53:37.296741 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:37.296874 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":304,"threads_started":1}
I20260621 02:53:37.358235 27132 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:37.358407 27132 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:37.358954 27132 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
W20260621 02:53:37.359920 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
W20260621 02:53:37.360167 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:37.360249 26741 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f55824adbd5645c5a14f5afa81f9c7bc; no voters: 44bc2988640b4c719d5b99cfa9601bcc, a457d954efb14a6c937bdeff7d4536d3
I20260621 02:53:37.360541 27132 raft_consensus.cc:2726] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:53:37.451736 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:37.450995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:44683)
0621 02:53:37.451457 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:37.451563 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260621 02:53:37.877009 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:37.876339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:36521)
0621 02:53:37.876769 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:37.876845 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260621 02:53:38.298671 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:38.297972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:58507)
0621 02:53:38.298413 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:38.298499 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":205,"threads_started":1}
I20260621 02:53:38.453517 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:38.452775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:51769)
0621 02:53:38.453211 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:38.453339 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260621 02:53:38.878541 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:38.877904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:43073)
0621 02:53:38.878302 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:38.878375 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260621 02:53:39.300163 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:39.299500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:49211)
0621 02:53:39.299902 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:39.299977 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":224,"threads_started":1}
I20260621 02:53:39.455054 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:39.454316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:50531)
0621 02:53:39.454744 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:39.454822 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":240,"threads_started":1}
I20260621 02:53:39.879938 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:39.879284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:39933)
0621 02:53:39.879689 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:39.879764 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260621 02:53:40.301748 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:40.301001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:38369)
0621 02:53:40.301492 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:40.301579 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":305,"threads_started":1}
I20260621 02:53:40.372565 27142 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:40.372740 27142 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:40.373353 27142 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
W20260621 02:53:40.374264 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
W20260621 02:53:40.374482 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:40.374608 26741 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f55824adbd5645c5a14f5afa81f9c7bc; no voters: 44bc2988640b4c719d5b99cfa9601bcc, a457d954efb14a6c937bdeff7d4536d3
I20260621 02:53:40.374900 27142 raft_consensus.cc:2726] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:53:40.456606 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:40.455884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:40129)
0621 02:53:40.456310 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:40.456401 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":234,"threads_started":1}
I20260621 02:53:40.881584 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:40.880789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:50905)
0621 02:53:40.881299 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:40.881394 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":218,"threads_started":1}
I20260621 02:53:41.303321 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:41.302593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:59379)
0621 02:53:41.303063 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:41.303148 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":208,"threads_started":1}
I20260621 02:53:41.458088 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:41.457421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:55581)
0621 02:53:41.457838 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:41.457921 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":215,"threads_started":1}
I20260621 02:53:41.883075 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:41.882419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:51953)
0621 02:53:41.882835 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:41.882914 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":216,"threads_started":1}
I20260621 02:53:42.304798 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:42.304183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:35143)
0621 02:53:42.304554 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:42.304641 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":197,"threads_started":1}
I20260621 02:53:42.459610 27149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:42.458922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:40855)
0621 02:53:42.459342 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:42.459416 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260621 02:53:42.884693 27150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:42.884033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:56251)
0621 02:53:42.884465 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:42.884544 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260621 02:53:43.306416 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:43.305685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:38873)
0621 02:53:43.306188 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:43.306271 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":232,"threads_started":1}
I20260621 02:53:43.461302 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:43.460514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:45745)
0621 02:53:43.460957 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:43.461048 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260621 02:53:43.886368 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:43.885663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:44543)
0621 02:53:43.886112 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:43.886201 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260621 02:53:44.308151 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:44.307355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:42243)
0621 02:53:44.307885 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:44.307964 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":326,"threads_started":1}
I20260621 02:53:44.462855 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:44.462189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:50351)
0621 02:53:44.462614 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:44.462691 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260621 02:53:44.887925 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:44.887263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:37701)
0621 02:53:44.887686 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:44.887762 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":202,"threads_started":1}
I20260621 02:53:45.057260 27157 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:45.057461 27157 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:45.058105 27157 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
W20260621 02:53:45.059008 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
W20260621 02:53:45.059244 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:45.059340 26741 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f55824adbd5645c5a14f5afa81f9c7bc; no voters: 44bc2988640b4c719d5b99cfa9601bcc, a457d954efb14a6c937bdeff7d4536d3
I20260621 02:53:45.059619 27157 raft_consensus.cc:2726] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:53:45.309875 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:45.308999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:44057)
0621 02:53:45.309636 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:45.309715 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":409,"threads_started":1}
I20260621 02:53:45.464588 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:45.463781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:40145)
0621 02:53:45.464263 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:45.464372 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":240,"threads_started":1}
I20260621 02:53:45.889540 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:45.888836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:45249)
0621 02:53:45.889300 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:45.889376 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":219,"threads_started":1}
I20260621 02:53:46.311542 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:46.310717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:41227)
0621 02:53:46.311293 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:46.311369 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":321,"threads_started":1}
I20260621 02:53:46.466221 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:46.465459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:48211)
0621 02:53:46.465908 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:46.465993 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260621 02:53:46.623374 27163 raft_consensus.cc:482] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:46.623610 27163 raft_consensus.cc:504] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a457d954efb14a6c937bdeff7d4536d3" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 35051 } } peers { permanent_uuid: "f55824adbd5645c5a14f5afa81f9c7bc" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45117 } } peers { permanent_uuid: "44bc2988640b4c719d5b99cfa9601bcc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 33029 } }
I20260621 02:53:46.624253 27163 leader_election.cc:288] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051), 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029)
W20260621 02:53:46.625262 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a457d954efb14a6c937bdeff7d4536d3 (127.11.109.3:35051): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
W20260621 02:53:46.625485 26741 leader_election.cc:341] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44bc2988640b4c719d5b99cfa9601bcc (127.11.109.2:33029): Not found: Tablet not found: 11b2d4b9c7a449dbbb5848b77926df4b
I20260621 02:53:46.625602 26741 leader_election.cc:302] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f55824adbd5645c5a14f5afa81f9c7bc; no voters: 44bc2988640b4c719d5b99cfa9601bcc, a457d954efb14a6c937bdeff7d4536d3
I20260621 02:53:46.625849 27163 raft_consensus.cc:2726] T 11b2d4b9c7a449dbbb5848b77926df4b P f55824adbd5645c5a14f5afa81f9c7bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:53:46.891229 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:46.890456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:33293)
0621 02:53:46.890914 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:46.891035 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":243,"threads_started":1}
I20260621 02:53:47.313162 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:47.312419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:37721)
0621 02:53:47.312913 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:47.313008 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
I20260621 02:53:47.467864 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:47.467135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:41941)
0621 02:53:47.467599 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:47.467688 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260621 02:53:47.892937 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:47.892187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:48699)
0621 02:53:47.892663 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:47.892751 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260621 02:53:48.314682 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:48.314011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:39283)
0621 02:53:48.314425 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:48.314503 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":232,"threads_started":1}
I20260621 02:53:48.469923 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:48.469046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.1:54529)
0621 02:53:48.469601 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:48.469717 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":292,"threads_started":1}
I20260621 02:53:48.894956 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:48.894107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.2:46149)
0621 02:53:48.894608 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:48.894783 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260621 02:53:49.297372 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26688
I20260621 02:53:49.312366 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:49.298358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.11.109.1:45117)
0621 02:53:49.298918 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:49.298927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:49.298981 (+ 54us) server_negotiation.cc:407] Connection header received
0621 02:53:49.299043 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:49.299050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:49.299325 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:49.299455 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:49.301792 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:49.302727 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:49.303873 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:49.304417 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:49.304510 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:49.307102 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:49.307112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:49.307125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:49.307158 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:49.309814 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:49.310580 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:49.310588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:49.310591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:49.310720 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:49.311032 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:49.311035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:49.311037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:49.311460 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:49.311681 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:49.311922 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 02:53:49.312102 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260621 02:53:49.313553 26847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:49.316648 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:49.315820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45657 (local address 127.11.109.3:52333)
0621 02:53:49.316274 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:49.316438 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
I20260621 02:53:49.402084 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26688
I20260621 02:53:49.420418 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26855
I20260621 02:53:49.433460 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:49.421154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.11.109.2:33029)
0621 02:53:49.421681 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:49.421686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:49.421699 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:53:49.421752 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:49.421755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:49.421821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:49.421937 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:49.423256 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:49.424017 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:49.425298 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:49.425731 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:49.425872 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:49.428353 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:49.428358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:49.428362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:49.428393 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:49.430823 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:49.431476 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:49.431483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:49.431487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:49.431619 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:49.431995 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:49.432000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:49.432004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:49.432266 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:49.432534 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:49.432820 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 02:53:49.433033 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":241,"threads_started":1}
I20260621 02:53:49.434158 26979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:49.516590 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26855
I20260621 02:53:49.533630 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 26987
I20260621 02:53:49.547639 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:49.534442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.11.109.3:35051)
0621 02:53:49.535193 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:49.535197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:49.535213 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:53:49.535267 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:49.535271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:49.535340 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:49.535461 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:49.536983 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:49.537928 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:49.539117 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:49.539565 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:49.539710 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:49.542281 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:49.542286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:49.542290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:49.542320 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:49.544944 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:49.545662 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:49.545667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:49.545676 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:49.545808 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:49.546201 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:49.546207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:49.546210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:49.546470 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:49.546757 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:49.547028 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 02:53:49.547235 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":465,"threads_started":1}
I20260621 02:53:49.548375 27110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:49.631431 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 26987
2026-06-21T02:53:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24107 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:53:49 dist-test-slave-bc52 krb5kdc[27180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:53:49 dist-test-slave-bc52 krb5kdc[27180](info): set up 2 sockets
Jun 21 02:53:49 dist-test-slave-bc52 krb5kdc[27180](info): commencing operation
krb5kdc: starting...
W20260621 02:53:50.717518 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:53:50 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:53:50.940659 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35281
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:51.257799 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:51.258118 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:51.258263 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:51.265093 27193 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:53:51.265242 27193 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:53:51.265334 27193 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:53:51.265437 27193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:53:51.265539 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:51.265683 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:51.265765 27193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:53:51.265846 27193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:53:51.275068 27193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:51.276247 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:51.277616 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:51.289515 27201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:51.289577 27199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:51.289741 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:51.290115 27193 server_base.cc:1034] running on GCE node
I20260621 02:53:51.290832 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:51.292088 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:51.293289 27193 hybrid_clock.cc:648] HybridClock initialized: now 1782010431293226 us; error 60 us; skew 500 ppm
I20260621 02:53:51.293596 27193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:51 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:51.296705 27193 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:53:51.298079 27193 webserver.cc:460] Webserver started at http://127.11.109.62:40389/ using document root <none> and password file <none>
I20260621 02:53:51.298743 27193 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:51.298873 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:51.299181 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:51.302074 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "8d6a497ee25a47c5999ad0a71dc25aee"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "14ae492bd29195a92fb54906b4e427ec"
server_key_iv: "d4831edd2bb49b85bd8fadb78d54882d"
server_key_version: "encryptionkey@0"
I20260621 02:53:51.302829 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "8d6a497ee25a47c5999ad0a71dc25aee"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "14ae492bd29195a92fb54906b4e427ec"
server_key_iv: "d4831edd2bb49b85bd8fadb78d54882d"
server_key_version: "encryptionkey@0"
I20260621 02:53:51.307586 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:53:51.310854 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:51.312069 27193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:53:51.312242 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "8d6a497ee25a47c5999ad0a71dc25aee"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "14ae492bd29195a92fb54906b4e427ec"
server_key_iv: "d4831edd2bb49b85bd8fadb78d54882d"
server_key_version: "encryptionkey@0"
I20260621 02:53:51.312450 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:51.326069 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:51.330456 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:51.330744 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:51.348790 27193 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:35281
I20260621 02:53:51.348767 27259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:35281 every 8 connection(s)
I20260621 02:53:51.350219 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:53:51.351758 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27193
I20260621 02:53:51.351970 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:53:51.352660 11700 external_mini_cluster.cc:1428] Setting key 3e846301f8bbbf83059f632c9ece0dc6
I20260621 02:53:51.354432 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:53:51 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:51.367498 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap starting.
I20260621 02:53:51.371455 27260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:51.372771 27260 log.cc:824] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:51.376173 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: No bootstrap required, opened a new log
I20260621 02:53:51.377199 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:51.354698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.11.109.62:35281)
0621 02:53:51.355456 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:51.355470 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:51.355517 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:53:51.357591 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:51.357617 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:51.358120 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:51.358524 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:51.359836 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:51.361871 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:51.363198 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:51.364224 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:51.364427 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:51.368542 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:51.368553 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:51.368570 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:51.368616 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:51.372554 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:51.373446 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:51.373455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:51.373469 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:51.373639 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:51.374074 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:51.374077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:51.374079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:51.374790 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:51.375082 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:51.375463 (+ 381us) server_negotiation.cc:299] Negotiation successful
0621 02:53:51.375883 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":313,"threads_started":1}
I20260621 02:53:51.384651 27260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:53:51.385000 27260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:51.385074 27260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6a497ee25a47c5999ad0a71dc25aee, State: Initialized, Role: FOLLOWER
I20260621 02:53:51.385663 27260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:53:51.385862 27260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:53:51.385962 27260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:53:51.386081 27260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:51.388484 27260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:53:51.388916 27260 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6a497ee25a47c5999ad0a71dc25aee; no voters:
I20260621 02:53:51.389514 27260 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:53:51.389669 27265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:51.390483 27265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 LEADER]: Becoming Leader. State: Replica: 8d6a497ee25a47c5999ad0a71dc25aee, State: Running, Role: LEADER
I20260621 02:53:51.391022 27265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:53:51.391696 27260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:53:51.393980 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6a497ee25a47c5999ad0a71dc25aee. Latest consensus state: current_term: 1 leader_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } } }
I20260621 02:53:51.393980 27266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } } }
I20260621 02:53:51.394488 27266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:51.394488 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: This master's current role is: LEADER
I20260621 02:53:51.395336 27273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:53:51.400239 27273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:53:51.409495 27273 catalog_manager.cc:1347] Generated new cluster ID: d2ae8e00383b4893ae3f8fb86b2a1027
I20260621 02:53:51.409615 27273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:53:51.424512 27273 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:53:51.426286 27273 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:53:51.435168 27273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Generated new TSK 0
I20260621 02:53:51.436103 27273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:53:51.494586 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35281
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:51.820170 27286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:51.820420 27286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:51.820513 27286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:51.827467 27286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:51.827574 27286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:51.827689 27286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:53:51.837025 27286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:51.838210 27286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:51.839414 27286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:51.851934 27291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:51.851886 27292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:51.852418 27294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:51.853129 27286 server_base.cc:1034] running on GCE node
I20260621 02:53:51.853809 27286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:51.854960 27286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:51.856201 27286 hybrid_clock.cc:648] HybridClock initialized: now 1782010431856144 us; error 50 us; skew 500 ppm
I20260621 02:53:51.856514 27286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:51 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:51.859566 27286 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:53:51.860841 27286 webserver.cc:460] Webserver started at http://127.11.109.1:39497/ using document root <none> and password file <none>
I20260621 02:53:51.861507 27286 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:51.861606 27286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:51.861909 27286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:51.864728 27286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "f1d2b966bb224840a189bda9fbfeaa0c"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "c80e5b7f5242806b7c0f89685c3aa044"
server_key_iv: "e5b0774dfcb2647a2c14758622c9a46a"
server_key_version: "encryptionkey@0"
I20260621 02:53:51.865489 27286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "f1d2b966bb224840a189bda9fbfeaa0c"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "c80e5b7f5242806b7c0f89685c3aa044"
server_key_iv: "e5b0774dfcb2647a2c14758622c9a46a"
server_key_version: "encryptionkey@0"
I20260621 02:53:51.870123 27286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:53:51.873255 27301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:51.874554 27286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:53:51.874732 27286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f1d2b966bb224840a189bda9fbfeaa0c"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "c80e5b7f5242806b7c0f89685c3aa044"
server_key_iv: "e5b0774dfcb2647a2c14758622c9a46a"
server_key_version: "encryptionkey@0"
I20260621 02:53:51.874917 27286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:51.888603 27286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:51.892576 27286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:51.892854 27286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:51.893815 27286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:51.895104 27286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:51.895203 27286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:51.895344 27286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:51.895418 27286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:51.928637 27286 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:34511
I20260621 02:53:51.928689 27413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:34511 every 8 connection(s)
I20260621 02:53:51.929949 27286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:53:51.935511 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27286
I20260621 02:53:51.935716 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:53:51.936473 11700 external_mini_cluster.cc:1428] Setting key e22471557868aa415625a34276108a6e
Jun 21 02:53:51 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:51.949565 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:51.932401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:41247 (local address 127.11.109.62:35281)
0621 02:53:51.932818 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:51.932823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:51.933831 (+ 1008us) server_negotiation.cc:407] Connection header received
0621 02:53:51.935836 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:51.935844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:51.935933 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:51.936059 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:51.937790 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:51.938678 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:51.940225 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:51.940615 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:51.940761 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:51.944089 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:51.944094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:51.944097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:51.944128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:51.946643 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:51.947343 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:51.947353 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:51.947355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:51.947468 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:51.947891 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:51.947896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:51.947899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:51.948146 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:51.948307 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:51.948969 (+ 662us) server_negotiation.cc:299] Negotiation successful
0621 02:53:51.949216 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260621 02:53:51.950448 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:51.932962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:41247)
0621 02:53:51.933660 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:51.933715 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:51.935526 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:51.936267 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:51.936276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:51.936825 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:51.937598 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:51.937625 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:51.938828 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:51.938832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:51.940060 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:51.940081 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:51.940847 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:51.940857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:51.941098 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:51.941803 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:51.941824 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:51.943882 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:51.946789 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:51.946799 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:51.946813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:51.947218 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:51.947578 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:51.947583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:51.947587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:51.947773 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:51.948276 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:51.948284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:51.948603 (+ 319us) client_negotiation.cc:769] Sending connection context
0621 02:53:51.948852 (+ 249us) client_negotiation.cc:240] Negotiation successful
0621 02:53:51.949361 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":298,"threads_started":1}
I20260621 02:53:51.951907 27414 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35281
I20260621 02:53:51.952222 27414 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:51.953073 27414 heartbeater.cc:511] Master 127.11.109.62:35281 requested a full tablet report, sending...
I20260621 02:53:51.955165 27225 ts_manager.cc:194] Registered new tserver with Master: f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1:34511)
I20260621 02:53:51.956557 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:41247
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:53:51.987709 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35281
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:52.303707 27420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:52.303956 27420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:52.304045 27420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:52.310832 27420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:52.310925 27420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:52.311028 27420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:53:52.319972 27420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:52.321055 27420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:52.322266 27420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:52.335219 27428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:52.335184 27425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:52.335294 27426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:52.341698 27420 server_base.cc:1034] running on GCE node
I20260621 02:53:52.342320 27420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:52.343253 27420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:52.344482 27420 hybrid_clock.cc:648] HybridClock initialized: now 1782010432344433 us; error 45 us; skew 500 ppm
I20260621 02:53:52.344789 27420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:52 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010432, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:52.348026 27420 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:53:52.350104 27420 webserver.cc:460] Webserver started at http://127.11.109.2:43495/ using document root <none> and password file <none>
I20260621 02:53:52.350781 27420 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:52.350922 27420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:52.351218 27420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:52.354030 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "dedcef199ac6465188f4007f4a6adad9"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "8a1d4f6e3ff2a44c7bb8bd0cf4782f30"
server_key_iv: "6383cf6ad68eee4cec5ea9c5413978f7"
server_key_version: "encryptionkey@0"
I20260621 02:53:52.354750 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "dedcef199ac6465188f4007f4a6adad9"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "8a1d4f6e3ff2a44c7bb8bd0cf4782f30"
server_key_iv: "6383cf6ad68eee4cec5ea9c5413978f7"
server_key_version: "encryptionkey@0"
I20260621 02:53:52.359788 27420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:53:52.363165 27435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:52.364416 27420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:53:52.364574 27420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "dedcef199ac6465188f4007f4a6adad9"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "8a1d4f6e3ff2a44c7bb8bd0cf4782f30"
server_key_iv: "6383cf6ad68eee4cec5ea9c5413978f7"
server_key_version: "encryptionkey@0"
I20260621 02:53:52.364774 27420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:52.380136 27420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:52.383827 27420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:52.384092 27420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:52.384953 27420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:52.386188 27420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:52.386308 27420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:52.386487 27420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:52.386555 27420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:52.418958 27420 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:44627
I20260621 02:53:52.419001 27547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:44627 every 8 connection(s)
I20260621 02:53:52.420174 27420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:53:52.427897 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27420
I20260621 02:53:52.428082 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:53:52.428639 11700 external_mini_cluster.cc:1428] Setting key a037654415d88e6651929726de52051a
Jun 21 02:53:52 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010432, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:52.440322 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.422570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:53349 (local address 127.11.109.62:35281)
0621 02:53:52.422750 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:52.422754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:52.423872 (+ 1118us) server_negotiation.cc:407] Connection header received
0621 02:53:52.425614 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:52.425619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:52.425683 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:52.425769 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:52.427096 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.427811 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.429949 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.430618 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.430751 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:52.434726 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:52.434731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:52.434734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:52.434766 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:52.437395 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:52.438138 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:52.438143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:52.438145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:52.438264 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:52.438692 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:52.438697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:52.438700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:52.438969 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:52.439114 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:52.439760 (+ 646us) server_negotiation.cc:299] Negotiation successful
0621 02:53:52.439978 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:53:52.441387 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.423063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:53349)
0621 02:53:52.423722 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:52.423758 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:52.425384 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:52.425964 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:52.425971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:52.426434 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:52.426959 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:52.426975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.427963 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.427969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:52.429816 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:52.429827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.430841 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.430848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:52.431124 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:52.431854 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:52.431887 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:52.434498 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:52.437539 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:52.437547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:52.437562 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:52.438008 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:52.438380 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:52.438386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:52.438390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:52.438578 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:52.439100 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:52.439108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:52.439419 (+ 311us) client_negotiation.cc:769] Sending connection context
0621 02:53:52.439667 (+ 248us) client_negotiation.cc:240] Negotiation successful
0621 02:53:52.440168 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
I20260621 02:53:52.442721 27548 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35281
I20260621 02:53:52.443038 27548 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:52.443876 27548 heartbeater.cc:511] Master 127.11.109.62:35281 requested a full tablet report, sending...
I20260621 02:53:52.445467 27225 ts_manager.cc:194] Registered new tserver with Master: dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627)
I20260621 02:53:52.446733 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:53349
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:53:52.482141 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35281
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:53:52.795703 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:53:52.795931 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:53:52.796015 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:53:52.803339 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:53:52.803431 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:53:52.803536 27553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:53:52.812620 27553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:53:52.813812 27553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:53:52.814997 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:53:52.826853 27558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:52.827040 27559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:53:52.828472 27561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:53:52.828866 27553 server_base.cc:1034] running on GCE node
I20260621 02:53:52.829470 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:53:52.830323 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:53:52.831513 27553 hybrid_clock.cc:648] HybridClock initialized: now 1782010432831473 us; error 36 us; skew 500 ppm
I20260621 02:53:52.831807 27553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:53:52 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010432, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:53:52.834774 27553 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:53:52.836185 27553 webserver.cc:460] Webserver started at http://127.11.109.3:41179/ using document root <none> and password file <none>
I20260621 02:53:52.836843 27553 fs_manager.cc:359] Metadata directory not provided
I20260621 02:53:52.836980 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:53:52.837329 27553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:53:52.840021 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "387dc238c3da47ff8a571588d1b5fb67"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "645b501de58aaae517fd7db41167bcc2"
server_key_iv: "6541c6ea081878e422c92168e985ac00"
server_key_version: "encryptionkey@0"
I20260621 02:53:52.840749 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "387dc238c3da47ff8a571588d1b5fb67"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "645b501de58aaae517fd7db41167bcc2"
server_key_iv: "6541c6ea081878e422c92168e985ac00"
server_key_version: "encryptionkey@0"
I20260621 02:53:52.845819 27553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:53:52.849115 27568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:52.850363 27553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260621 02:53:52.850525 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "387dc238c3da47ff8a571588d1b5fb67"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "645b501de58aaae517fd7db41167bcc2"
server_key_iv: "6541c6ea081878e422c92168e985ac00"
server_key_version: "encryptionkey@0"
I20260621 02:53:52.850711 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:53:52.876348 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:53:52.880499 27553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:53:52.880781 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:53:52.881738 27553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:53:52.882974 27553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:53:52.883072 27553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:52.883213 27553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:53:52.883282 27553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:53:52.915278 27553 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:43967
I20260621 02:53:52.915365 27680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:43967 every 8 connection(s)
I20260621 02:53:52.916541 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:53:52.922770 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27553
I20260621 02:53:52.922957 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:53:52.923548 11700 external_mini_cluster.cc:1428] Setting key 4e717a37cfa080cf3dd7579e3b4d96e8
Jun 21 02:53:52 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010432, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:53:52.936918 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.919037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:53759 (local address 127.11.109.62:35281)
0621 02:53:52.919241 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:52.919247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:52.920309 (+ 1062us) server_negotiation.cc:407] Connection header received
0621 02:53:52.922083 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:52.922088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:52.922178 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:52.922263 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:52.924230 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.925271 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.927058 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.927446 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.927540 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:52.931104 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:52.931109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:52.931114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:52.931149 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:52.933869 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:52.934552 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:52.934561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:52.934564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:52.934685 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:52.935099 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:52.935106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:52.935111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:52.935380 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:52.935531 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:52.936368 (+ 837us) server_negotiation.cc:299] Negotiation successful
0621 02:53:52.936580 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:53:52.938037 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.919497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:53759)
0621 02:53:52.920156 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:52.920192 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:52.921850 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:52.922512 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:52.922524 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:52.923247 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:53:52.924050 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:52.924076 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.925421 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.925426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:52.926933 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:52.926948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.927617 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.927621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:52.927805 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:52.928465 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:53:52.928493 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:53:52.930911 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:53:52.934013 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:52.934021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:52.934035 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:52.934423 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:52.934808 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:53:52.934813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:53:52.934817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:53:52.934989 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:53:52.935501 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:53:52.935508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:53:52.935910 (+ 402us) client_negotiation.cc:769] Sending connection context
0621 02:53:52.936262 (+ 352us) client_negotiation.cc:240] Negotiation successful
0621 02:53:52.936756 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260621 02:53:52.939332 27681 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35281
I20260621 02:53:52.939575 27681 heartbeater.cc:464] Registering TS with master...
I20260621 02:53:52.940184 27681 heartbeater.cc:511] Master 127.11.109.62:35281 requested a full tablet report, sending...
I20260621 02:53:52.941469 27225 ts_manager.cc:194] Registered new tserver with Master: 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
I20260621 02:53:52.942276 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:53759
I20260621 02:53:52.950716 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:53:52.959672 27414 heartbeater.cc:503] Master 127.11.109.62:35281 was elected leader, sending a full tablet report...
I20260621 02:53:52.967325 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.955075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.11.109.62:35281)
0621 02:53:52.955303 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:52.955312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:52.955689 (+ 377us) server_negotiation.cc:407] Connection header received
0621 02:53:52.955894 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:52.955900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:52.955979 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:52.956075 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:52.957489 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.958249 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.959758 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:52.960343 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:52.960460 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:52.962907 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:52.962911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:52.962914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:52.962942 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:52.965248 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:52.965824 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:52.965828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:52.965830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:52.965954 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:52.966253 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:52.966256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:52.966258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:52.966551 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:52.966717 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:52.966971 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 02:53:52.967119 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:53:52.971185 27225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:53:52.977833 27225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:53:53.009567 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.988919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:44627 (local address 127.0.0.1:33398)
0621 02:53:52.990543 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:52.990596 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:52.990796 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:52.999539 (+ 8743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:52.999547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:52.999585 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:53.000005 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.000018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.002383 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.002392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.005569 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.005589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.008096 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.008106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.008429 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.008509 (+ 80us) client_negotiation.cc:769] Sending connection context
0621 02:53:53.008642 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 02:53:53.008906 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":3,"thread_start_us":10458,"threads_started":1}
I20260621 02:53:53.011216 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.989107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.11.109.2:44627)
0621 02:53:52.998881 (+ 9774us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:52.998893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:52.998929 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:53:52.999035 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:52.999046 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:52.999310 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:52.999529 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:53.000178 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.002192 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.005848 (+ 3656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.007815 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.010205 (+ 2390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.010477 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.010559 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.010789 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9573,"thread_start_us":9474,"threads_started":1}
I20260621 02:53:53.013830 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.989719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:43967 (local address 127.0.0.1:33066)
0621 02:53:52.999656 (+ 9937us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:52.999675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:52.999804 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:53.000652 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:53.000659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:53.000694 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:53.001209 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.001220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.005113 (+ 3893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.005121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.009359 (+ 4238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.009383 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.012282 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.012288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.012480 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.012504 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:53:53.012599 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:53:53.013463 (+ 864us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9783,"mutex_wait_us":29,"thread_start_us":1032,"threads_started":1}
I20260621 02:53:53.015453 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:52.990448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.11.109.3:43967)
0621 02:53:52.999303 (+ 8855us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:52.999310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:52.999796 (+ 486us) server_negotiation.cc:407] Connection header received
0621 02:53:52.999988 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:52.999995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.000227 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.000400 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:53.001607 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.004896 (+ 3289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.009588 (+ 4692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.011714 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.011891 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.012135 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.012888 (+ 753us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.013052 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8682,"thread_start_us":8527,"threads_started":1}
I20260621 02:53:53.015743 27483 tablet_service.cc:1467] Processing CreateTablet for tablet 0a061d57b6504ad9a836632c441d3164 (DEFAULT_TABLE table=test-table [id=fb2c77b6cc8540a9afaa0f3a6367541c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:53.017062 27483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a061d57b6504ad9a836632c441d3164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:53.018564 27616 tablet_service.cc:1467] Processing CreateTablet for tablet 0a061d57b6504ad9a836632c441d3164 (DEFAULT_TABLE table=test-table [id=fb2c77b6cc8540a9afaa0f3a6367541c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:53.019641 27616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a061d57b6504ad9a836632c441d3164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:53.022158 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.000306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:59092)
0621 02:53:53.009861 (+ 9555us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:53.009883 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:53.010000 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:53.012583 (+ 2583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:53.012591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:53.012617 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:53.012811 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.012821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.014583 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.014590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.015987 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.015998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.021399 (+ 5401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.021405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.021569 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.021631 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 02:53:53.021715 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 02:53:53.021863 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9418,"thread_start_us":11691,"threads_started":1}
I20260621 02:53:53.023067 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.000017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.11.109.1:34511)
0621 02:53:53.012023 (+ 12006us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.012032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.012060 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:53:53.012155 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.012164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.012387 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.012582 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:53.012982 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.014446 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.016125 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.021230 (+ 5105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.021390 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.022532 (+ 1142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.022600 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.022756 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11825,"thread_start_us":11657,"threads_started":1}
I20260621 02:53:53.026453 27349 tablet_service.cc:1467] Processing CreateTablet for tablet 0a061d57b6504ad9a836632c441d3164 (DEFAULT_TABLE table=test-table [id=fb2c77b6cc8540a9afaa0f3a6367541c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:53:53.027791 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a061d57b6504ad9a836632c441d3164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:53:53.028780 27699 tablet_bootstrap.cc:492] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Bootstrap starting.
I20260621 02:53:53.031615 27699 tablet_bootstrap.cc:654] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:53.032752 27699 log.cc:824] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:53.033017 27700 tablet_bootstrap.cc:492] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Bootstrap starting.
I20260621 02:53:53.037048 27700 tablet_bootstrap.cc:654] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:53.038327 27700 log.cc:824] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:53.038869 27699 tablet_bootstrap.cc:492] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: No bootstrap required, opened a new log
I20260621 02:53:53.039885 27699 ts_tablet_manager.cc:1397] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260621 02:53:53.042726 27703 tablet_bootstrap.cc:492] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Bootstrap starting.
I20260621 02:53:53.043884 27700 tablet_bootstrap.cc:492] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: No bootstrap required, opened a new log
I20260621 02:53:53.044169 27700 ts_tablet_manager.cc:1397] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260621 02:53:53.046164 27703 tablet_bootstrap.cc:654] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Neither blocks nor log segments found. Creating new log.
I20260621 02:53:53.047329 27703 log.cc:824] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:53:53.047983 27699 raft_consensus.cc:348] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.048467 27699 raft_consensus.cc:374] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:53.048573 27699 raft_consensus.cc:729] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedcef199ac6465188f4007f4a6adad9, State: Initialized, Role: FOLLOWER
I20260621 02:53:53.049165 27699 consensus_queue.cc:260] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.050230 27703 tablet_bootstrap.cc:492] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: No bootstrap required, opened a new log
I20260621 02:53:53.050477 27703 ts_tablet_manager.cc:1397] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:53:53.053885 27548 heartbeater.cc:503] Master 127.11.109.62:35281 was elected leader, sending a full tablet report...
I20260621 02:53:53.054781 27699 ts_tablet_manager.cc:1428] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260621 02:53:53.054697 27700 raft_consensus.cc:348] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.055120 27700 raft_consensus.cc:374] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:53.055258 27700 raft_consensus.cc:729] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387dc238c3da47ff8a571588d1b5fb67, State: Initialized, Role: FOLLOWER
I20260621 02:53:53.055806 27700 consensus_queue.cc:260] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.056886 27681 heartbeater.cc:503] Master 127.11.109.62:35281 was elected leader, sending a full tablet report...
I20260621 02:53:53.057494 27700 ts_tablet_manager.cc:1428] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260621 02:53:53.060470 27703 raft_consensus.cc:348] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.060897 27703 raft_consensus.cc:374] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:53:53.061015 27703 raft_consensus.cc:729] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d2b966bb224840a189bda9fbfeaa0c, State: Initialized, Role: FOLLOWER
I20260621 02:53:53.061575 27703 consensus_queue.cc:260] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.063098 27703 ts_tablet_manager.cc:1428] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260621 02:53:53.168387 27682 tablet.cc:2369] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:53:53.172015 27549 tablet.cc:2369] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:53.174719 27707 raft_consensus.cc:482] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:53:53.174999 27707 raft_consensus.cc:504] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.176571 27707 leader_election.cc:288] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
W20260621 02:53:53.182157 27415 tablet.cc:2369] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:53:53.183835 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.177139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:46413 (local address 127.11.109.2:44627)
0621 02:53:53.177375 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.177383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.177960 (+ 577us) server_negotiation.cc:407] Connection header received
0621 02:53:53.178295 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.178301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.178403 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.178493 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:53.179086 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.179962 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.181972 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.182940 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.183111 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.183181 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.183439 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.183559 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 02:53:53.183969 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.177045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:44627 (local address 127.11.109.1:46413)
0621 02:53:53.177814 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:53.177837 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:53.178085 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:53.178511 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:53.178516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:53.178547 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:53.178958 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.178973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.180092 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.180098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.181828 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.181839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.183134 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.183140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.183288 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.183311 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:53:53.183414 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:53:53.183545 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":53,"thread_start_us":522,"threads_started":1}
I20260621 02:53:53.184254 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.177047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:46925 (local address 127.11.109.3:43967)
0621 02:53:53.177276 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.177282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.178412 (+ 1130us) server_negotiation.cc:407] Connection header received
0621 02:53:53.178638 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.178643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.178709 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.178786 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:53:53.179572 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.180460 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.182431 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.183328 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.183497 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.183573 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.183874 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.183999 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:53:53.184535 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.177375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:43967 (local address 127.11.109.1:46925)
0621 02:53:53.178304 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0621 02:53:53.178320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:53:53.178440 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:53:53.179024 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:53:53.179028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:53:53.179056 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:53:53.179406 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.179419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.180598 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.180602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.182232 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:53:53.182246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.183544 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.183551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:53:53.183718 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.183744 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 02:53:53.183848 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 02:53:53.183974 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":37,"thread_start_us":640,"threads_started":1}
I20260621 02:53:53.184933 27503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a061d57b6504ad9a836632c441d3164" candidate_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dedcef199ac6465188f4007f4a6adad9" is_pre_election: true
I20260621 02:53:53.185580 27503 raft_consensus.cc:2454] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d2b966bb224840a189bda9fbfeaa0c in term 0.
I20260621 02:53:53.185580 27636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a061d57b6504ad9a836632c441d3164" candidate_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387dc238c3da47ff8a571588d1b5fb67" is_pre_election: true
I20260621 02:53:53.186076 27636 raft_consensus.cc:2454] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d2b966bb224840a189bda9fbfeaa0c in term 0.
I20260621 02:53:53.186318 27304 leader_election.cc:302] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dedcef199ac6465188f4007f4a6adad9, f1d2b966bb224840a189bda9fbfeaa0c; no voters:
I20260621 02:53:53.186731 27707 raft_consensus.cc:2781] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:53:53.186875 27707 raft_consensus.cc:482] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:53:53.186987 27707 raft_consensus.cc:3037] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:53.188995 27707 raft_consensus.cc:504] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.189668 27707 leader_election.cc:288] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 1 election: Requested vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
I20260621 02:53:53.190035 27503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a061d57b6504ad9a836632c441d3164" candidate_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dedcef199ac6465188f4007f4a6adad9"
I20260621 02:53:53.190142 27636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a061d57b6504ad9a836632c441d3164" candidate_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "387dc238c3da47ff8a571588d1b5fb67"
I20260621 02:53:53.190250 27503 raft_consensus.cc:3037] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:53.190337 27636 raft_consensus.cc:3037] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:53:53.193454 27503 raft_consensus.cc:2454] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d2b966bb224840a189bda9fbfeaa0c in term 1.
I20260621 02:53:53.193454 27636 raft_consensus.cc:2454] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d2b966bb224840a189bda9fbfeaa0c in term 1.
I20260621 02:53:53.193833 27304 leader_election.cc:302] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dedcef199ac6465188f4007f4a6adad9, f1d2b966bb224840a189bda9fbfeaa0c; no voters:
I20260621 02:53:53.194092 27707 raft_consensus.cc:2781] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:53:53.194526 27707 raft_consensus.cc:686] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 LEADER]: Becoming Leader. State: Replica: f1d2b966bb224840a189bda9fbfeaa0c, State: Running, Role: LEADER
I20260621 02:53:53.194963 27707 consensus_queue.cc:237] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } }
I20260621 02:53:53.199079 27224 catalog_manager.cc:5495] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1). New cstate: current_term: 1 leader_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } health_report { overall_health: UNKNOWN } } }
I20260621 02:53:53.235384 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.230675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.11.109.1:34511)
0621 02:53:53.230856 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.230860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.230876 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:53:53.230959 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.230962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.231044 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.231165 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:53:53.231492 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.232259 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.233584 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.233953 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.234038 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.234487 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:53:53.234957 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:53:53.235060 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.235144 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.235201 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:53:53.243355 27636 raft_consensus.cc:1261] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Refusing update from remote peer f1d2b966bb224840a189bda9fbfeaa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:53.243428 27503 raft_consensus.cc:1261] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Refusing update from remote peer f1d2b966bb224840a189bda9fbfeaa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:53:53.244314 27710 consensus_queue.cc:1035] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:53.244668 27707 consensus_queue.cc:1035] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:53:53.255906 27713 mvcc.cc:205] Tried to move back new op lower bound from 7299114734554030080 to 7299114734369914880. Current Snapshot: MvccSnapshot[applied={T|T < 7299114734554030080}]
I20260621 02:53:53.269440 27225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:53.269716 27225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:53:53.273030 27225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Sending DeleteTablet for 3 replicas of tablet 0a061d57b6504ad9a836632c441d3164
I20260621 02:53:53.274282 27483 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a061d57b6504ad9a836632c441d3164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:53 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:33398
I20260621 02:53:53.274315 27616 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a061d57b6504ad9a836632c441d3164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:53 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:33066
I20260621 02:53:53.274317 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a061d57b6504ad9a836632c441d3164 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:53:53 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:59092
I20260621 02:53:53.277367 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27286
I20260621 02:53:53.287770 27723 tablet_replica.cc:330] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: stopping tablet replica
I20260621 02:53:53.288189 27723 raft_consensus.cc:2229] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:53.288705 27723 raft_consensus.cc:2258] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:53.289749 27725 tablet_replica.cc:330] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: stopping tablet replica
I20260621 02:53:53.289964 27724 tablet_replica.cc:330] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: stopping tablet replica
I20260621 02:53:53.290169 27725 raft_consensus.cc:2229] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:53:53.290350 27724 raft_consensus.cc:2229] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:53:53.290613 27725 raft_consensus.cc:2258] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:53.290963 27724 raft_consensus.cc:2258] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:53:53.292654 27725 ts_tablet_manager.cc:1905] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:53.293300 27724 ts_tablet_manager.cc:1905] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:53:53.295809 27723 ts_tablet_manager.cc:1905] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:53:53 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:53:53.299893 27725 ts_tablet_manager.cc:1918] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:53.300042 27725 log.cc:1196] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/0a061d57b6504ad9a836632c441d3164
I20260621 02:53:53.300632 27725 ts_tablet_manager.cc:1939] T 0a061d57b6504ad9a836632c441d3164 P dedcef199ac6465188f4007f4a6adad9: Deleting consensus metadata
I20260621 02:53:53.300731 27724 ts_tablet_manager.cc:1918] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:53.300943 27724 log.cc:1196] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/0a061d57b6504ad9a836632c441d3164
I20260621 02:53:53.301499 27724 ts_tablet_manager.cc:1939] T 0a061d57b6504ad9a836632c441d3164 P f1d2b966bb224840a189bda9fbfeaa0c: Deleting consensus metadata
I20260621 02:53:53.302455 27211 catalog_manager.cc:4841] TS dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): tablet 0a061d57b6504ad9a836632c441d3164 (table test-table [id=fb2c77b6cc8540a9afaa0f3a6367541c]) successfully deleted
I20260621 02:53:53.303228 27211 catalog_manager.cc:4841] TS f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1:34511): tablet 0a061d57b6504ad9a836632c441d3164 (table test-table [id=fb2c77b6cc8540a9afaa0f3a6367541c]) successfully deleted
I20260621 02:53:53.305676 27723 ts_tablet_manager.cc:1918] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:53:53.305847 27723 log.cc:1196] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/0a061d57b6504ad9a836632c441d3164
I20260621 02:53:53.306339 27723 ts_tablet_manager.cc:1939] T 0a061d57b6504ad9a836632c441d3164 P 387dc238c3da47ff8a571588d1b5fb67: Deleting consensus metadata
I20260621 02:53:53.307647 27209 catalog_manager.cc:4841] TS 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): tablet 0a061d57b6504ad9a836632c441d3164 (table test-table [id=fb2c77b6cc8540a9afaa0f3a6367541c]) successfully deleted
I20260621 02:53:53.312860 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.278612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.11.109.1:34511)
0621 02:53:53.278848 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.278857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.285471 (+ 6614us) server_negotiation.cc:407] Connection header received
0621 02:53:53.286537 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.286546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.286645 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.286747 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:53.290257 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.291490 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.293738 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.294475 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.294572 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.306541 (+ 11969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:53.306556 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:53.306573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:53.306618 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:53.309941 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:53.310854 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:53.310865 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:53.310867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:53.311008 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:53.311426 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:53.311429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:53.311431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:53.311769 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:53.311976 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.312358 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.312536 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:53:53.313498 27409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:53.412456 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27286
I20260621 02:53:53.431686 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27420
Jun 21 02:53:53 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:53:53.446913 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.432545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.11.109.2:44627)
0621 02:53:53.432923 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.432928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.432946 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:53:53.432994 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.432997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.433052 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.433163 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:53.434471 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.435251 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.436695 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.437094 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.437297 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.441673 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:53.441685 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:53.441704 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:53.441736 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:53.444511 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:53.445153 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:53.445160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:53.445162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:53.445333 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:53.445637 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:53.445640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:53.445642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:53.445992 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:53.446261 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.446521 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.446668 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260621 02:53:53.447660 27543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:53.536502 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27420
I20260621 02:53:53.553411 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27553
Jun 21 02:53:53 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:53:53.568436 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:53:53.554146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.11.109.3:43967)
0621 02:53:53.554317 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:53:53.554321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:53:53.554361 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:53:53.554507 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:53:53.554511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:53:53.554574 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:53:53.554650 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:53:53.555998 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.556705 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.558042 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:53:53.558432 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:53:53.558560 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:53:53.562898 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:53:53.562913 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:53:53.562928 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:53:53.562966 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:53:53.565670 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:53.566393 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:53.566409 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:53.566411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:53.566544 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:53:53.566913 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:53:53.566919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:53:53.566922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:53:53.567268 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:53:53.567581 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 02:53:53.567900 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 02:53:53.568107 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:53:53.569083 27676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:53.655442 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27553
I20260621 02:53:53.671708 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27193
I20260621 02:53:53.672787 27255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:53:53.779788 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27193
I20260621 02:54:13.804330 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35281
--webserver_interface=127.11.109.62
--webserver_port=40389
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:54:14.128463 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:14.128715 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:14.128882 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:14.135850 27731 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:54:14.135932 27731 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:54:14.136023 27731 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:54:14.136133 27731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:54:14.136224 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:14.136323 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:14.136409 27731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:54:14.136487 27731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:54:14.145833 27731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:14.146984 27731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:14.148239 27731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:14.173677 27740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:14.173782 27738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:14.173786 27737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:14.174607 27731 server_base.cc:1034] running on GCE node
I20260621 02:54:14.175530 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:14.177143 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:14.178435 27731 hybrid_clock.cc:648] HybridClock initialized: now 1782010454178371 us; error 27 us; skew 500 ppm
I20260621 02:54:14.178865 27731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:14 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:14.187021 27731 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:54:14.188635 27731 webserver.cc:460] Webserver started at http://127.11.109.62:40389/ using document root <none> and password file <none>
I20260621 02:54:14.189424 27731 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:14.189525 27731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:14.195346 27731 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 02:54:14.198230 27747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:14.199520 27731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:54:14.199702 27731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "8d6a497ee25a47c5999ad0a71dc25aee"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "14ae492bd29195a92fb54906b4e427ec"
server_key_iv: "d4831edd2bb49b85bd8fadb78d54882d"
server_key_version: "encryptionkey@0"
I20260621 02:54:14.200702 27731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:14.214066 27731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:14.218384 27731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:14.218678 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:14.239166 27731 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:35281
I20260621 02:54:14.239197 27798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:35281 every 8 connection(s)
I20260621 02:54:14.240658 27731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:54:14.246623 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27731
I20260621 02:54:14.247439 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:34511
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:35281
--webserver_port=39497
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:54:14.248622 27799 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:54:14.252437 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap starting.
I20260621 02:54:14.281139 27799 log.cc:824] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:14.296535 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:54:14.297202 27799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap complete.
I20260621 02:54:14.309125 27799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:14.310009 27799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6a497ee25a47c5999ad0a71dc25aee, State: Initialized, Role: FOLLOWER
I20260621 02:54:14.310644 27799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:14.310895 27799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:54:14.311026 27799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:54:14.311195 27799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:54:14.314889 27799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:14.315382 27799 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6a497ee25a47c5999ad0a71dc25aee; no voters:
I20260621 02:54:14.315982 27799 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:54:14.316165 27803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:54:14.317299 27803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 LEADER]: Becoming Leader. State: Replica: 8d6a497ee25a47c5999ad0a71dc25aee, State: Running, Role: LEADER
I20260621 02:54:14.318035 27799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:54:14.317900 27803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:14.320327 27805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6a497ee25a47c5999ad0a71dc25aee. Latest consensus state: current_term: 2 leader_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } } }
I20260621 02:54:14.320688 27805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:14.321156 27804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } } }
I20260621 02:54:14.321434 27804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:14.330338 27808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:54:14.340050 27808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:54:14.341899 27808 catalog_manager.cc:1259] Loaded cluster ID: d2ae8e00383b4893ae3f8fb86b2a1027
I20260621 02:54:14.342084 27808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:54:14.347223 27808 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:54:14.349341 27808 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Loaded TSK: 0
I20260621 02:54:14.359352 27820 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:54:14.377067 27808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Generated new TSK 1
I20260621 02:54:14.377429 27808 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:54:14.619328 27801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:14.619580 27801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:14.619725 27801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:14.626895 27801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:14.626999 27801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:14.627169 27801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:54:14.636917 27801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:34511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=39497
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:14.638172 27801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:14.639436 27801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:14.653088 27826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:14.653053 27827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:14.653667 27829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:14.655216 27801 server_base.cc:1034] running on GCE node
I20260621 02:54:14.655821 27801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:14.656818 27801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:14.658031 27801 hybrid_clock.cc:648] HybridClock initialized: now 1782010454657981 us; error 66 us; skew 500 ppm
I20260621 02:54:14.658339 27801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:14 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:14.661772 27801 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:54:14.663509 27801 webserver.cc:460] Webserver started at http://127.11.109.1:39497/ using document root <none> and password file <none>
I20260621 02:54:14.664453 27801 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:14.664609 27801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:14.671440 27801 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:54:14.674733 27836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:14.676335 27801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:54:14.676491 27801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f1d2b966bb224840a189bda9fbfeaa0c"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "c80e5b7f5242806b7c0f89685c3aa044"
server_key_iv: "e5b0774dfcb2647a2c14758622c9a46a"
server_key_version: "encryptionkey@0"
I20260621 02:54:14.677824 27801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:14.694787 27801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:14.699683 27801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:14.699939 27801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:14.701045 27801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:14.702729 27801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:14.702863 27801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:14.702982 27801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:14.703064 27801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:14.742269 27801 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:34511
I20260621 02:54:14.742306 27948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:34511 every 8 connection(s)
I20260621 02:54:14.743821 27801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:54:14.745584 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27801
I20260621 02:54:14.746315 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:44627
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:35281
--webserver_port=43495
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:54:14 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:14.771440 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:14.747061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:44365)
0621 02:54:14.749886 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:14.749938 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:14.751699 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:14.753443 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:14.753457 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:14.753945 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:14.754484 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:14.754502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:14.756806 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:14.756809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:14.758096 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:14.758108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:14.759513 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:14.759518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:14.759692 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:14.760381 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:14.760402 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:14.762567 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:14.767468 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:14.767474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:14.767485 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:14.767924 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:14.768461 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:14.768464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:14.768467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:14.768628 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:14.769610 (+ 982us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:14.769615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:14.769876 (+ 261us) client_negotiation.cc:769] Sending connection context
0621 02:54:14.770094 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 02:54:14.770474 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2542,"thread_start_us":2318,"threads_started":1}
I20260621 02:54:14.772174 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:14.746931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:44365 (local address 127.11.109.62:35281)
0621 02:54:14.747663 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:14.747684 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:14.750088 (+ 2404us) server_negotiation.cc:407] Connection header received
0621 02:54:14.752283 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:14.752304 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:14.752806 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:14.753264 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:14.754678 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:14.756646 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:14.758238 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:14.759301 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:14.759479 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:14.762866 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:14.762879 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:14.762901 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:14.762957 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:14.767324 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:14.768099 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:14.768109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:14.768118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:14.768341 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:14.768746 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:14.768754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:14.768759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:14.769482 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:14.769686 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:14.770395 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 02:54:14.770869 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":270,"threads_started":1}
I20260621 02:54:14.774878 27949 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35281
I20260621 02:54:14.775202 27949 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:14.775980 27949 heartbeater.cc:511] Master 127.11.109.62:35281 requested a full tablet report, sending...
I20260621 02:54:14.778136 27764 ts_manager.cc:194] Registered new tserver with Master: f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1:34511)
I20260621 02:54:14.779698 27764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:44365
W20260621 02:54:15.069908 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:15.070185 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:15.070415 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:15.077420 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:15.077524 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:15.077721 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:54:15.086913 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:44627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=43495
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:15.088043 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:15.089373 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:15.101951 27961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:15.101982 27958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:15.102072 27959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:15.103154 27953 server_base.cc:1034] running on GCE node
I20260621 02:54:15.103762 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:15.104823 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:15.106030 27953 hybrid_clock.cc:648] HybridClock initialized: now 1782010455105997 us; error 30 us; skew 500 ppm
I20260621 02:54:15.106345 27953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:15 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:15.109447 27953 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:54:15.110759 27953 webserver.cc:460] Webserver started at http://127.11.109.2:43495/ using document root <none> and password file <none>
I20260621 02:54:15.111370 27953 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:15.111498 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:15.117787 27953 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:54:15.120457 27968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:15.122072 27953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:15.122228 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "dedcef199ac6465188f4007f4a6adad9"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "8a1d4f6e3ff2a44c7bb8bd0cf4782f30"
server_key_iv: "6383cf6ad68eee4cec5ea9c5413978f7"
server_key_version: "encryptionkey@0"
I20260621 02:54:15.123169 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:15.151762 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:15.156111 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:15.156381 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:15.157378 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:15.158658 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:15.158778 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:15.158908 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:15.159003 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:15.191391 27953 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:44627
I20260621 02:54:15.191442 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:44627 every 8 connection(s)
I20260621 02:54:15.192838 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:54:15.200649 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 27953
I20260621 02:54:15.201419 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:43967
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:35281
--webserver_port=41179
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:54:15 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:15.213356 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.195445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:46787 (local address 127.11.109.62:35281)
0621 02:54:15.195605 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.195609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.196697 (+ 1088us) server_negotiation.cc:407] Connection header received
0621 02:54:15.198472 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.198476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.198552 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.198636 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:15.199971 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.200649 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.203155 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.203581 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.203682 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.207782 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:15.207787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:15.207791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:15.207822 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:15.210434 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.211142 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.211152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.211154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.211271 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.211702 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.211707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.211711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.211994 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:15.212155 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.212770 (+ 615us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.213003 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:54:15.214208 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.195849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:46787)
0621 02:54:15.196531 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.196570 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.198230 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.198836 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.198843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.199310 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:15.199813 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.199828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.200801 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.200806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.202972 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.203001 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.203802 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.203811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.204160 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.204903 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:15.204935 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:15.207552 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:15.210589 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:15.210598 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:15.210613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:15.211013 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:15.211390 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:15.211394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:15.211398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:15.211582 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:15.212123 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:15.212132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:15.212423 (+ 291us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.212677 (+ 254us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.213137 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
I20260621 02:54:15.215682 28081 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35281
I20260621 02:54:15.216002 28081 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:15.216660 28081 heartbeater.cc:511] Master 127.11.109.62:35281 requested a full tablet report, sending...
I20260621 02:54:15.218098 27764 ts_manager.cc:194] Registered new tserver with Master: dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627)
I20260621 02:54:15.218916 27764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:46787
W20260621 02:54:15.520406 28084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:15.520651 28084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:15.520805 28084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:15.528043 28084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:15.528141 28084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:15.528362 28084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:54:15.537905 28084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:43967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=41179
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:15.539117 28084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:15.540349 28084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:15.553243 28089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:15.553215 28090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:15.553202 28092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:15.554369 28084 server_base.cc:1034] running on GCE node
I20260621 02:54:15.554932 28084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:15.555835 28084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:15.557068 28084 hybrid_clock.cc:648] HybridClock initialized: now 1782010455557034 us; error 56 us; skew 500 ppm
I20260621 02:54:15.557401 28084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:15 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:15.560359 28084 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:54:15.561807 28084 webserver.cc:460] Webserver started at http://127.11.109.3:41179/ using document root <none> and password file <none>
I20260621 02:54:15.562451 28084 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:15.562605 28084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:15.567952 28084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:54:15.570571 28099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:15.571795 28084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:54:15.571957 28084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "387dc238c3da47ff8a571588d1b5fb67"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "645b501de58aaae517fd7db41167bcc2"
server_key_iv: "6541c6ea081878e422c92168e985ac00"
server_key_version: "encryptionkey@0"
I20260621 02:54:15.572880 28084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:15.594177 28084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:15.597980 28084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:15.598259 28084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:15.599205 28084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:15.600459 28084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:15.600576 28084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:15.600723 28084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:15.600809 28084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:15.634462 28084 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:43967
I20260621 02:54:15.634544 28211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:43967 every 8 connection(s)
I20260621 02:54:15.635871 28084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:54:15.645255 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28084
Jun 21 02:54:15 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:15.656050 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.638318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:45783 (local address 127.11.109.62:35281)
0621 02:54:15.638532 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.638538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.639739 (+ 1201us) server_negotiation.cc:407] Connection header received
0621 02:54:15.641483 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.641488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.641549 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.641632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:15.643047 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.643739 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.645420 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.645993 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.646137 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.650105 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:15.650114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:15.650117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:15.650147 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:15.653139 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.653886 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.653891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.653895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.654019 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.654425 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.654430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.654433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.654696 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:15.654853 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.655475 (+ 622us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.655690 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:54:15.656920 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.638887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:45783)
0621 02:54:15.639583 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.639627 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.641258 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.641864 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.641872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.642360 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:15.642899 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.642915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.643904 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.643908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.645269 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.645281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.646245 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.646252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.646621 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.647433 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:15.647458 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:15.649888 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:15.653360 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:15.653374 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:15.653385 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:15.653760 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:15.654125 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:15.654130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:15.654136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:15.654308 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:15.654828 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:15.654836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:15.655129 (+ 293us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.655383 (+ 254us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.655852 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":261,"threads_started":1}
I20260621 02:54:15.658308 28212 heartbeater.cc:347] Connected to a master server at 127.11.109.62:35281
I20260621 02:54:15.658593 28212 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:15.659428 28212 heartbeater.cc:511] Master 127.11.109.62:35281 requested a full tablet report, sending...
I20260621 02:54:15.660745 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.646227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.11.109.62:35281)
0621 02:54:15.647373 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.647377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.647392 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:54:15.647428 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.647431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.647486 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.647613 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:15.649521 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.650280 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.651633 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.652170 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.652310 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.655240 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:15.655246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:15.655251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:15.655288 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:15.658186 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.658917 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.658922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.658925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.659059 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.659406 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.659412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.659417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.659759 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:15.660013 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.660205 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.660379 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":886,"threads_started":1}
I20260621 02:54:15.661085 27764 ts_manager.cc:194] Registered new tserver with Master: 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
I20260621 02:54:15.662094 27764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:45783
I20260621 02:54:15.663314 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:54:15.669399 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.664262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.11.109.62:35281)
0621 02:54:15.664410 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.664415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.665027 (+ 612us) server_negotiation.cc:407] Connection header received
0621 02:54:15.665222 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.665229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.665357 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.665466 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:54:15.665860 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.666495 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.667817 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.668220 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.668322 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.668741 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:54:15.668916 (+ 175us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 02:54:15.669158 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56798: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:54:15.669517 28216 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56798: token expired
W20260621 02:54:15.670046 27687 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.109.62:35281)
I20260621 02:54:15.681510 28216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.670454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.11.109.62:35281)
0621 02:54:15.670598 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.670602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.670618 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:54:15.670743 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.670747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.670791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.670860 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:15.672094 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.672782 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.674179 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.674543 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.674634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.677158 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:15.677163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:15.677166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:15.677241 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:15.679456 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.679997 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.680000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.680002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.680109 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.680400 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.680403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.680405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.680661 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:15.680820 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.681078 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.681297 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 02:54:15.684676 27763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:54:15.691810 27763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:54:15.731472 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.702786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.11.109.2:44627)
0621 02:54:15.723028 (+ 20242us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.723038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.723069 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:54:15.723157 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.723163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.723396 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.724231 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:15.724435 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.725986 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.729335 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.730314 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.730448 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.730646 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.731025 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.731202 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20077,"thread_start_us":21806,"threads_started":1}
I20260621 02:54:15.732328 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.702482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:44627 (local address 127.0.0.1:50904)
0621 02:54:15.717828 (+ 15346us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.717880 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.718099 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.723615 (+ 5516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.723621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.723652 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:15.724032 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.724046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.726214 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.726220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.729108 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.729137 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.730553 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.730559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.730772 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.730832 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.730950 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.731151 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15042,"mutex_wait_us":246,"thread_start_us":15834,"threads_started":1}
I20260621 02:54:15.732363 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.703030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.11.109.3:43967)
0621 02:54:15.721661 (+ 18631us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.721673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.721713 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:54:15.721828 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.721839 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.722153 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.722402 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:15.723362 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.725797 (+ 2435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.729279 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.730657 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.730823 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.731051 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.731851 (+ 800us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.732006 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18407,"thread_start_us":18195,"threads_started":1}
I20260621 02:54:15.732918 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.719201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41250)
0621 02:54:15.722845 (+ 3644us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.722863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.722981 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.725716 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.725720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.725742 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:15.726171 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.726182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.727977 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.727982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.730111 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.730126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.731884 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.731890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.732043 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.732075 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.732123 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.732283 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3508,"mutex_wait_us":80,"thread_start_us":4058,"threads_started":1}
I20260621 02:54:15.733021 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.702742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:43967 (local address 127.0.0.1:39990)
0621 02:54:15.720331 (+ 17589us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.720376 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.720530 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.722448 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.722459 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.722509 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:15.723087 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.723103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.726002 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.726007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.729109 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.729143 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.731198 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.731203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.731356 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.731383 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.731468 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.731570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17436,"mutex_wait_us":151,"thread_start_us":17876,"threads_started":1}
I20260621 02:54:15.733302 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.719485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.11.109.1:34511)
0621 02:54:15.725143 (+ 5658us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.725151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.725200 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:54:15.725299 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.725305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.725515 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.726365 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:15.726596 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.727822 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.730281 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.731661 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.732435 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.732701 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.732765 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.732942 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5503,"thread_start_us":5444,"threads_started":1}
I20260621 02:54:15.735839 28016 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef052f363b044c8961234aa187c9169 (DEFAULT_TABLE table=test-table [id=9fc776d239ef4e6d836eb87452e23c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:15.736227 28147 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef052f363b044c8961234aa187c9169 (DEFAULT_TABLE table=test-table [id=9fc776d239ef4e6d836eb87452e23c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:15.737221 28016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef052f363b044c8961234aa187c9169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:15.737617 28147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef052f363b044c8961234aa187c9169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:15.738812 27884 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef052f363b044c8961234aa187c9169 (DEFAULT_TABLE table=test-table [id=9fc776d239ef4e6d836eb87452e23c0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:15.740056 27884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef052f363b044c8961234aa187c9169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:15.756105 28229 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Bootstrap starting.
I20260621 02:54:15.759719 28228 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Bootstrap starting.
I20260621 02:54:15.761865 28230 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Bootstrap starting.
I20260621 02:54:15.762283 28229 tablet_bootstrap.cc:654] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:15.762862 28228 tablet_bootstrap.cc:654] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:15.763381 28229 log.cc:824] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:15.764094 28228 log.cc:824] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:15.765460 28230 tablet_bootstrap.cc:654] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:15.766261 28229 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: No bootstrap required, opened a new log
I20260621 02:54:15.766574 28229 ts_tablet_manager.cc:1397] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260621 02:54:15.767015 28230 log.cc:824] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:15.767762 28228 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: No bootstrap required, opened a new log
I20260621 02:54:15.768008 28228 ts_tablet_manager.cc:1397] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:54:15.769496 28230 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: No bootstrap required, opened a new log
I20260621 02:54:15.769734 28230 ts_tablet_manager.cc:1397] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:54:15.776446 28230 raft_consensus.cc:348] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.776968 28230 raft_consensus.cc:374] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:15.777119 28230 raft_consensus.cc:729] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d2b966bb224840a189bda9fbfeaa0c, State: Initialized, Role: FOLLOWER
I20260621 02:54:15.777030 28229 raft_consensus.cc:348] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.777535 28229 raft_consensus.cc:374] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:15.777683 28229 raft_consensus.cc:729] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedcef199ac6465188f4007f4a6adad9, State: Initialized, Role: FOLLOWER
I20260621 02:54:15.777988 28230 consensus_queue.cc:260] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.778240 28229 consensus_queue.cc:260] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.778266 28228 raft_consensus.cc:348] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.778664 28228 raft_consensus.cc:374] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:15.778813 28228 raft_consensus.cc:729] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387dc238c3da47ff8a571588d1b5fb67, State: Initialized, Role: FOLLOWER
I20260621 02:54:15.779042 27949 heartbeater.cc:503] Master 127.11.109.62:35281 was elected leader, sending a full tablet report...
I20260621 02:54:15.779486 28228 consensus_queue.cc:260] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.784024 28212 heartbeater.cc:503] Master 127.11.109.62:35281 was elected leader, sending a full tablet report...
I20260621 02:54:15.784441 28228 ts_tablet_manager.cc:1428] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Time spent starting tablet: real 0.016s user 0.013s sys 0.005s
I20260621 02:54:15.784693 28230 ts_tablet_manager.cc:1428] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20260621 02:54:15.785056 28081 heartbeater.cc:503] Master 127.11.109.62:35281 was elected leader, sending a full tablet report...
I20260621 02:54:15.786315 28229 ts_tablet_manager.cc:1428] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260621 02:54:15.795014 28236 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:15.795246 28236 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.796665 28236 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1:34511)
I20260621 02:54:15.802380 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.797032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:57331 (local address 127.11.109.2:44627)
0621 02:54:15.797267 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.797274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.797366 (+ 92us) server_negotiation.cc:407] Connection header received
0621 02:54:15.797632 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.797638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.797725 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.797852 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:15.798263 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.799002 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.800595 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.801459 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.801615 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.801698 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.802012 (+ 314us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.802131 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:54:15.802395 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.797018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:44627 (local address 127.11.109.3:57331)
0621 02:54:15.797264 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.797284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.797458 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.797867 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.797872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.797900 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:15.798078 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.798127 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.799141 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.799146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.800430 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.800442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.801607 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.801613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.801856 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.801881 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.801987 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.802118 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":44}
I20260621 02:54:15.803658 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.797683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:35465 (local address 127.11.109.1:34511)
0621 02:54:15.797872 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.797877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.798607 (+ 730us) server_negotiation.cc:407] Connection header received
0621 02:54:15.798790 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.798795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.798859 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.798946 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:15.799512 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.800193 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.801865 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.802841 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.802987 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.803056 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.803392 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.803481 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:54:15.803540 28036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef052f363b044c8961234aa187c9169" candidate_uuid: "387dc238c3da47ff8a571588d1b5fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dedcef199ac6465188f4007f4a6adad9" is_pre_election: true
I20260621 02:54:15.803877 28237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.797465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.11.109.3:35465)
0621 02:54:15.798506 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:15.798520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:15.798663 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:15.798960 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:15.798964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:15.798988 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:15.799348 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.799356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.800322 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.800327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.801723 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:15.801735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.803019 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.803044 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:15.803223 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.803259 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:54:15.803352 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:54:15.803490 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":21,"thread_start_us":724,"threads_started":1}
I20260621 02:54:15.804100 28036 raft_consensus.cc:2454] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387dc238c3da47ff8a571588d1b5fb67 in term 0.
I20260621 02:54:15.804911 28102 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9; no voters:
I20260621 02:54:15.804869 27904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef052f363b044c8961234aa187c9169" candidate_uuid: "387dc238c3da47ff8a571588d1b5fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" is_pre_election: true
I20260621 02:54:15.805300 28236 raft_consensus.cc:2781] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:54:15.805368 27904 raft_consensus.cc:2454] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 387dc238c3da47ff8a571588d1b5fb67 in term 0.
I20260621 02:54:15.805454 28236 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:54:15.805575 28236 raft_consensus.cc:3037] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:15.807842 28236 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.808470 28236 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [CANDIDATE]: Term 1 election: Requested vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1:34511)
I20260621 02:54:15.808770 28036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef052f363b044c8961234aa187c9169" candidate_uuid: "387dc238c3da47ff8a571588d1b5fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dedcef199ac6465188f4007f4a6adad9"
I20260621 02:54:15.809005 28036 raft_consensus.cc:3037] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:15.808991 27904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef052f363b044c8961234aa187c9169" candidate_uuid: "387dc238c3da47ff8a571588d1b5fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d2b966bb224840a189bda9fbfeaa0c"
I20260621 02:54:15.809170 27904 raft_consensus.cc:3037] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:15.811134 28036 raft_consensus.cc:2454] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387dc238c3da47ff8a571588d1b5fb67 in term 1.
I20260621 02:54:15.811305 27904 raft_consensus.cc:2454] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 387dc238c3da47ff8a571588d1b5fb67 in term 1.
I20260621 02:54:15.811493 28102 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9; no voters:
I20260621 02:54:15.811755 28236 raft_consensus.cc:2781] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:54:15.812223 28236 raft_consensus.cc:686] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 LEADER]: Becoming Leader. State: Replica: 387dc238c3da47ff8a571588d1b5fb67, State: Running, Role: LEADER
I20260621 02:54:15.812654 28236 consensus_queue.cc:237] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:15.817148 27762 catalog_manager.cc:5495] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "387dc238c3da47ff8a571588d1b5fb67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } health_report { overall_health: UNKNOWN } } }
I20260621 02:54:15.837116 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.831984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.11.109.3:43967)
0621 02:54:15.832196 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.832201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.832431 (+ 230us) server_negotiation.cc:407] Connection header received
0621 02:54:15.832595 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.832598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.832674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.832786 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:54:15.833093 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.833842 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.835146 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.835530 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.835616 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.836070 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:54:15.836622 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:54:15.836733 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.836876 (+ 143us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.836937 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:54:15.844748 28036 raft_consensus.cc:1261] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Refusing update from remote peer 387dc238c3da47ff8a571588d1b5fb67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:54:15.844748 27904 raft_consensus.cc:1261] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Refusing update from remote peer 387dc238c3da47ff8a571588d1b5fb67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:54:15.845726 28238 consensus_queue.cc:1035] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:54:15.846073 28236 consensus_queue.cc:1035] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:54:15.856582 28241 mvcc.cc:205] Tried to move back new op lower bound from 7299114827130007552 to 7299114827012599808. Current Snapshot: MvccSnapshot[applied={T|T < 7299114827130007552}]
I20260621 02:54:15.858013 28242 mvcc.cc:205] Tried to move back new op lower bound from 7299114827130007552 to 7299114827012599808. Current Snapshot: MvccSnapshot[applied={T|T < 7299114827130007552}]
I20260621 02:54:15.868127 27762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:54:15.868379 27762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:54:15.871734 27762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Sending DeleteTablet for 3 replicas of tablet 4ef052f363b044c8961234aa187c9169
I20260621 02:54:15.872807 28016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef052f363b044c8961234aa187c9169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:54:15 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:50904
I20260621 02:54:15.872867 28147 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef052f363b044c8961234aa187c9169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:54:15 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:39990
I20260621 02:54:15.873154 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27801
I20260621 02:54:15.873842 27884 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef052f363b044c8961234aa187c9169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:54:15 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:41250
I20260621 02:54:15.885222 28250 tablet_replica.cc:330] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: stopping tablet replica
I20260621 02:54:15.885759 28251 tablet_replica.cc:330] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: stopping tablet replica
I20260621 02:54:15.886288 28251 raft_consensus.cc:2229] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:54:15.886804 28250 raft_consensus.cc:2229] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:54:15.887390 28251 raft_consensus.cc:2258] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:54:15.888073 28250 raft_consensus.cc:2258] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:54:15.889463 28252 tablet_replica.cc:330] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: stopping tablet replica
I20260621 02:54:15.890488 28251 ts_tablet_manager.cc:1905] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:54:15.890807 28250 ts_tablet_manager.cc:1905] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:54:15.894323 28252 raft_consensus.cc:2229] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:54:15.894933 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:15.874107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.11.109.1:34511)
0621 02:54:15.874312 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:15.874321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:15.874370 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:54:15.874590 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:15.874598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:15.874753 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:15.874883 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:15.877003 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.878544 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.881033 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:15.882026 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:15.882172 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:15.886417 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:15.886433 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:15.886449 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:15.886506 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:15.890843 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.891698 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.891706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.891709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.891842 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:15.892260 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:15.892282 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:15.892285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:15.892748 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:15.893010 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:15.894322 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0621 02:54:15.894555 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:54:15.895929 27944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:15.897851 28250 ts_tablet_manager.cc:1918] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:54:15.898083 28250 log.cc:1196] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/4ef052f363b044c8961234aa187c9169
I20260621 02:54:15.898200 28251 ts_tablet_manager.cc:1918] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:54:15.898486 28251 log.cc:1196] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/4ef052f363b044c8961234aa187c9169
I20260621 02:54:15.898650 28250 ts_tablet_manager.cc:1939] T 4ef052f363b044c8961234aa187c9169 P 387dc238c3da47ff8a571588d1b5fb67: Deleting consensus metadata
I20260621 02:54:15.899194 28251 ts_tablet_manager.cc:1939] T 4ef052f363b044c8961234aa187c9169 P dedcef199ac6465188f4007f4a6adad9: Deleting consensus metadata
I20260621 02:54:15.900414 27748 catalog_manager.cc:4841] TS 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): tablet 4ef052f363b044c8961234aa187c9169 (table test-table [id=9fc776d239ef4e6d836eb87452e23c0e]) successfully deleted
I20260621 02:54:15.901214 27750 catalog_manager.cc:4841] TS dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): tablet 4ef052f363b044c8961234aa187c9169 (table test-table [id=9fc776d239ef4e6d836eb87452e23c0e]) successfully deleted
I20260621 02:54:15.993525 28252 raft_consensus.cc:2258] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:54:15.994179 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27801
I20260621 02:54:16.011655 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27953
W20260621 02:54:16.011802 27750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.109.1:34511 (error 108)
I20260621 02:54:16.013971 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.013490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41262)
0621 02:54:16.013677 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.013794 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 02:54:16.014276 27750 catalog_manager.cc:4584] TS f1d2b966bb224840a189bda9fbfeaa0c (127.11.109.1:34511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ef052f363b044c8961234aa187c9169: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
I20260621 02:54:16.030259 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.012438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.11.109.2:44627)
0621 02:54:16.012625 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:16.012642 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:16.012666 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:54:16.012925 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:16.012933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:16.013023 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:16.013120 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:16.015128 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:16.015844 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:16.017244 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:16.017633 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:16.017764 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:16.024203 (+ 6439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:16.024224 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:16.024240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:16.024285 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:16.027294 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:16.028116 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:16.028131 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:16.028134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:16.028252 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:16.028644 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:16.028650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:16.028653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:16.029011 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:16.029342 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:16.029665 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 02:54:16.029874 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:54:16.030893 28076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:16.067744 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.067281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41278)
0621 02:54:16.067507 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.067587 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260621 02:54:16.068786 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.068437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41290)
0621 02:54:16.068602 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.068682 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:54:16.129571 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.129042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41294)
0621 02:54:16.129314 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.129400 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 02:54:16.130703 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.130312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41304)
0621 02:54:16.130467 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.130547 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260621 02:54:16.179816 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27953
I20260621 02:54:16.196872 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 28084
I20260621 02:54:16.211175 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.210622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41308)
0621 02:54:16.210869 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.211027 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260621 02:54:16.212479 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.211960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:34511 (local address 127.0.0.1:41312)
0621 02:54:16.212198 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:16.212322 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:34511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260621 02:54:16.214834 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:16.197688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.11.109.3:43967)
0621 02:54:16.197867 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:16.197871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:16.197951 (+ 80us) server_negotiation.cc:407] Connection header received
0621 02:54:16.198156 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:16.198160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:16.198246 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:16.198325 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:16.200174 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:16.201097 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:16.203118 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:16.203610 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:16.203711 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:16.207282 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:16.207293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:16.207309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:16.207354 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:16.211310 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:16.212337 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:16.212347 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:16.212351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:16.212600 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:16.213068 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:16.213074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:16.213078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:16.213613 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:16.213931 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:16.214200 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 02:54:16.214427 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:54:16.215664 28207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:16.340400 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 28084
I20260621 02:54:16.360080 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 27731
I20260621 02:54:16.361131 27794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:16.474812 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 27731
I20260621 02:54:36.496651 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35281
--webserver_interface=127.11.109.62
--webserver_port=40389
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:54:36.818580 28258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:36.818903 28258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:36.819064 28258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:36.826256 28258 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:54:36.826385 28258 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:54:36.826478 28258 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:54:36.826572 28258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:54:36.826670 28258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:36.826790 28258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:36.826876 28258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:54:36.826960 28258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:54:36.836488 28258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=40389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:36.837811 28258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:36.839087 28258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:36.850553 28264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:36.850625 28263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:36.850625 28266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:36.851773 28258 server_base.cc:1034] running on GCE node
I20260621 02:54:36.852509 28258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:36.853953 28258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:36.855190 28258 hybrid_clock.cc:648] HybridClock initialized: now 1782010476855131 us; error 39 us; skew 500 ppm
I20260621 02:54:36.855511 28258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:36 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010476, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:36.862023 28258 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:54:36.863492 28258 webserver.cc:460] Webserver started at http://127.11.109.62:40389/ using document root <none> and password file <none>
I20260621 02:54:36.864162 28258 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:36.864312 28258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:36.869606 28258 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:36.872444 28273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:36.873805 28258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:36.873986 28258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "8d6a497ee25a47c5999ad0a71dc25aee"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "14ae492bd29195a92fb54906b4e427ec"
server_key_iv: "d4831edd2bb49b85bd8fadb78d54882d"
server_key_version: "encryptionkey@0"
I20260621 02:54:36.875010 28258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:36.887166 28258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:36.891055 28258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:36.891337 28258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:36.909384 28258 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:35281
I20260621 02:54:36.909407 28324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:35281 every 8 connection(s)
I20260621 02:54:36.910816 28258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:54:36.917294 28325 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:54:36.919458 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28258
I20260621 02:54:36.920508 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:34511
--local_ip_for_outbound_sockets=127.11.109.1
--tserver_master_addrs=127.11.109.62:35281
--webserver_port=39497
--webserver_interface=127.11.109.1
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:54:36.921067 28325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap starting.
I20260621 02:54:36.942853 28325 log.cc:824] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:36.958361 28325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:54:36.958899 28325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Bootstrap complete.
I20260621 02:54:36.967734 28325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:36.968588 28325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6a497ee25a47c5999ad0a71dc25aee, State: Initialized, Role: FOLLOWER
I20260621 02:54:36.969209 28325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:36.969470 28325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:54:36.969594 28325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:54:36.969792 28325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:54:36.973758 28325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:36.974244 28325 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6a497ee25a47c5999ad0a71dc25aee; no voters:
I20260621 02:54:36.974944 28325 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 02:54:36.975078 28329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:54:36.976298 28329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [term 3 LEADER]: Becoming Leader. State: Replica: 8d6a497ee25a47c5999ad0a71dc25aee, State: Running, Role: LEADER
I20260621 02:54:36.976900 28329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } }
I20260621 02:54:36.977118 28325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:54:36.985586 28331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6a497ee25a47c5999ad0a71dc25aee. Latest consensus state: current_term: 3 leader_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } } }
I20260621 02:54:36.985932 28331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:36.995872 28330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6a497ee25a47c5999ad0a71dc25aee" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35281 } } }
I20260621 02:54:36.996222 28330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:36.996882 28338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:54:37.007301 28338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:54:37.009352 28338 catalog_manager.cc:1259] Loaded cluster ID: d2ae8e00383b4893ae3f8fb86b2a1027
I20260621 02:54:37.009518 28338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:54:37.014868 28338 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:54:37.017078 28338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Loaded TSK: 0
I20260621 02:54:37.017367 28338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d6a497ee25a47c5999ad0a71dc25aee: Loaded TSK: 1
F20260621 02:54:37.018901 28338 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe3e23c29ad google::LogMessage::Fail() at ??:0
@ 0x7fe3e23c588a google::LogMessage::SendToLog() at ??:0
@ 0x7fe3e23c24de google::LogMessage::Flush() at ??:0
@ 0x7fe3e23c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe3e61267d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe3e611573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe3f7df11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe3f7df0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe3f7e7c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe3f7e7c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe3f7ec65e9 std::function<>::operator()() at ??:0
@ 0x7fe3f7df4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe3f7df2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe3f7e7a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe3f7e7a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe3e500e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe3e501191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe3e5011799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe3e4feceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe3e82c66db start_thread at ??:0
@ 0x7fe3e125c61f clone at ??:0
W20260621 02:54:37.308450 28327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:37.308703 28327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:37.308853 28327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:37.315905 28327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:37.316002 28327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:37.316202 28327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:54:37.325426 28327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:34511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=39497
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:37.326565 28327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:37.327893 28327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:37.341073 28372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:37.340986 28375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:37.341041 28373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:37.341902 28327 server_base.cc:1034] running on GCE node
I20260621 02:54:37.342447 28327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:37.343281 28327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:37.344514 28327 hybrid_clock.cc:648] HybridClock initialized: now 1782010477344471 us; error 53 us; skew 500 ppm
I20260621 02:54:37.344828 28327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:37 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010477, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:37.347921 28327 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:54:37.349527 28327 webserver.cc:460] Webserver started at http://127.11.109.1:39497/ using document root <none> and password file <none>
I20260621 02:54:37.350232 28327 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:37.350376 28327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:37.355692 28327 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:54:37.358407 28382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:37.359707 28327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:37.359876 28327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "f1d2b966bb224840a189bda9fbfeaa0c"
format_stamp: "Formatted at 2026-06-21 02:53:51 on dist-test-slave-bc52"
server_key: "c80e5b7f5242806b7c0f89685c3aa044"
server_key_iv: "e5b0774dfcb2647a2c14758622c9a46a"
server_key_version: "encryptionkey@0"
I20260621 02:54:37.360800 28327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:37.387643 28327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:37.391857 28327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:37.392127 28327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:37.393258 28327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:37.395066 28389 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260621 02:54:37.400295 28327 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260621 02:54:37.400439 28327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260621 02:54:37.400576 28327 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260621 02:54:37.403640 28327 ts_tablet_manager.cc:610] Registered 1 tablets
I20260621 02:54:37.403743 28327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260621 02:54:37.404054 28389 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Bootstrap starting.
I20260621 02:54:37.429695 28389 log.cc:824] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:37.445749 28389 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:54:37.446287 28389 tablet_bootstrap.cc:492] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Bootstrap complete.
I20260621 02:54:37.447530 28389 ts_tablet_manager.cc:1397] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.010s
I20260621 02:54:37.451669 28327 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:34511
I20260621 02:54:37.451836 28496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:34511 every 8 connection(s)
I20260621 02:54:37.452903 28327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:54:37.456358 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28327
I20260621 02:54:37.455653 28389 raft_consensus.cc:348] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:37.457134 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:44627
--local_ip_for_outbound_sockets=127.11.109.2
--tserver_master_addrs=127.11.109.62:35281
--webserver_port=43495
--webserver_interface=127.11.109.2
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:54:37.457085 28389 raft_consensus.cc:729] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d2b966bb224840a189bda9fbfeaa0c, State: Initialized, Role: FOLLOWER
I20260621 02:54:37.458022 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.455697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:59361)
0621 02:54:37.456649 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.456969 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":432,"threads_started":1}
I20260621 02:54:37.457895 28389 consensus_queue.cc:260] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
W20260621 02:54:37.458630 28497 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:35281 (0 consecutive failures): Network error: Failed to ping master at 127.11.109.62:35281: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
I20260621 02:54:37.459640 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.459222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:43871)
0621 02:54:37.459449 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.459533 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260621 02:54:37.460217 28389 ts_tablet_manager.cc:1428] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260621 02:54:37.460534 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.460155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:39951)
0621 02:54:37.460332 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.460421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:54:37.460783 28497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:54:37.461550 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.461115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:42995)
0621 02:54:37.461319 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.461408 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:54:37.778087 28501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:37.778402 28501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:37.778568 28501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:37.789403 28501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:37.789542 28501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:37.789734 28501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:54:37.799338 28501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:44627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=43495
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:37.800582 28501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:37.801935 28501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:37.814692 28506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:37.814730 28507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:37.814769 28509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:37.815687 28501 server_base.cc:1034] running on GCE node
I20260621 02:54:37.816262 28501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:37.817134 28501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:37.818382 28501 hybrid_clock.cc:648] HybridClock initialized: now 1782010477818314 us; error 62 us; skew 500 ppm
I20260621 02:54:37.818873 28501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:37 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010477, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:37.822032 28501 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:54:37.823397 28501 webserver.cc:460] Webserver started at http://127.11.109.2:43495/ using document root <none> and password file <none>
I20260621 02:54:37.824047 28501 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:37.824193 28501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:37.829335 28501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:54:37.833693 28516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:37.835047 28501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260621 02:54:37.835222 28501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "dedcef199ac6465188f4007f4a6adad9"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "8a1d4f6e3ff2a44c7bb8bd0cf4782f30"
server_key_iv: "6383cf6ad68eee4cec5ea9c5413978f7"
server_key_version: "encryptionkey@0"
I20260621 02:54:37.836181 28501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:37.854295 28501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:37.858227 28501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:37.858495 28501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:37.859447 28501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:37.860740 28501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:37.860867 28501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:37.860996 28501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:37.861083 28501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:37.893628 28501 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:44627
I20260621 02:54:37.893713 28628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:44627 every 8 connection(s)
I20260621 02:54:37.895102 28501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:54:37.900061 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.898201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:51503)
0621 02:54:37.898963 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.899222 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":277,"threads_started":1}
W20260621 02:54:37.900743 28629 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:35281 (0 consecutive failures): Network error: Failed to ping master at 127.11.109.62:35281: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
I20260621 02:54:37.901443 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28501
I20260621 02:54:37.901672 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.901331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:56817)
0621 02:54:37.901471 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.901537 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:54:37.902133 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:43967
--local_ip_for_outbound_sockets=127.11.109.3
--tserver_master_addrs=127.11.109.62:35281
--webserver_port=41179
--webserver_interface=127.11.109.3
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:54:37.902501 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:37.902162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:51525)
0621 02:54:37.902301 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:37.902385 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:54:37.902726 28629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:54:38.219066 28632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:38.219311 28632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:38.219417 28632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:38.227494 28632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:38.227609 28632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:38.227731 28632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:54:38.238303 28632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:43967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=41179
--tserver_master_addrs=127.11.109.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:38.239606 28632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:38.241022 28632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:38.255209 28640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:38.255244 28638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:38.255273 28637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:38.256076 28632 server_base.cc:1034] running on GCE node
I20260621 02:54:38.256696 28632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:38.257637 28632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:38.258850 28632 hybrid_clock.cc:648] HybridClock initialized: now 1782010478258806 us; error 60 us; skew 500 ppm
I20260621 02:54:38.259153 28632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:38 dist-test-slave-bc52 krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010478, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:38.262040 28632 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:54:38.263403 28632 webserver.cc:460] Webserver started at http://127.11.109.3:41179/ using document root <none> and password file <none>
I20260621 02:54:38.264094 28632 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:38.264195 28632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:38.269435 28632 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:54:38.271914 28647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:38.273280 28632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:38.273443 28632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "387dc238c3da47ff8a571588d1b5fb67"
format_stamp: "Formatted at 2026-06-21 02:53:52 on dist-test-slave-bc52"
server_key: "645b501de58aaae517fd7db41167bcc2"
server_key_iv: "6541c6ea081878e422c92168e985ac00"
server_key_version: "encryptionkey@0"
I20260621 02:54:38.274380 28632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:38.298835 28632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:38.302613 28632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:38.302896 28632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:38.303768 28632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:38.304975 28632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:38.305064 28632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:38.305140 28632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:38.305215 28632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:38.336947 28632 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:43967
I20260621 02:54:38.337025 28759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:43967 every 8 connection(s)
I20260621 02:54:38.338346 28632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:54:38.343029 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.341370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:33365)
0621 02:54:38.342053 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.342284 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
W20260621 02:54:38.343658 28760 heartbeater.cc:644] Failed to heartbeat to 127.11.109.62:35281 (0 consecutive failures): Network error: Failed to ping master at 127.11.109.62:35281: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
I20260621 02:54:38.343683 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28632
I20260621 02:54:38.344421 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.344106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:46549)
0621 02:54:38.344251 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.344327 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:54:38.345466 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.345073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:45199)
0621 02:54:38.345233 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.345325 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:54:38.345705 28760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:54:38.463047 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.462287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:53717)
0621 02:54:38.462754 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.462834 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":218,"threads_started":1}
I20260621 02:54:38.901705 28765 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:38.902082 28765 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:38.904136 28765 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
I20260621 02:54:38.919387 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.903334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:38745)
0621 02:54:38.918857 (+ 15523us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.919065 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15390,"threads_started":1}
Jun 21 02:54:38 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010477, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:54:38.931661 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.904469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:44627 (local address 127.11.109.1:46585)
0621 02:54:38.904691 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.904746 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:38.905348 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:38.910804 (+ 5456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:38.910819 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:38.911790 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:38.912854 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:38.912886 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.916576 (+ 3690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.916585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:38.918513 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:38.918525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.919565 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.919571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:38.919856 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:38.920835 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:38.920892 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:38.923973 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:38.927699 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:38.927708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:38.927722 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:38.928331 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:38.928854 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:38.928859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:38.928863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:38.929107 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:38.930225 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:38.930231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:38.930524 (+ 293us) client_negotiation.cc:769] Sending connection context
0621 02:54:38.930783 (+ 259us) client_negotiation.cc:240] Negotiation successful
0621 02:54:38.931248 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"spinlock_wait_cycles":11136}
I20260621 02:54:38.931717 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.905755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:46585 (local address 127.11.109.2:44627)
0621 02:54:38.906370 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:38.906389 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:38.906444 (+ 55us) server_negotiation.cc:407] Connection header received
0621 02:54:38.909576 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:38.909591 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:38.910156 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:38.913216 (+ 3060us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:38.913840 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.916289 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.918670 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.919375 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.919571 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:38.924244 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:38.924253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:38.924269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:38.924323 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:38.927542 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:38.928518 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:38.928526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:38.928533 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:38.928732 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:38.929277 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:38.929286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:38.929292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:38.930065 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:38.930270 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:38.931016 (+ 746us) server_negotiation.cc:299] Negotiation successful
0621 02:54:38.931352 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
Jun 21 02:54:38 dist-test-slave-bc52 krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010477, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
W20260621 02:54:38.935364 28385 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
I20260621 02:54:38.941604 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.905475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:43967 (local address 127.11.109.1:39555)
0621 02:54:38.917753 (+ 12278us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:38.917782 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:38.917929 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:38.921194 (+ 3265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:38.921199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:38.921795 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:38.922100 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:38.922108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.925445 (+ 3337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.925452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:38.927168 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:38.927186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.928551 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.928560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:38.928769 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:38.930447 (+ 1678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:38.930503 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:38.933099 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:38.938200 (+ 5101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:38.938206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:38.938209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:38.938746 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:38.939284 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:38.939291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:38.939296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:38.939490 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:38.940408 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:38.940415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:38.940589 (+ 174us) client_negotiation.cc:769] Sending connection context
0621 02:54:38.940746 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 02:54:38.941034 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12100,"spinlock_wait_cycles":25856,"thread_start_us":12542,"threads_started":1}
I20260621 02:54:38.941857 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:38.905296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:39555 (local address 127.11.109.3:43967)
0621 02:54:38.917722 (+ 12426us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:38.917735 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:38.918320 (+ 585us) server_negotiation.cc:407] Connection header received
0621 02:54:38.920384 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:38.920397 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:38.920847 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:38.921213 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:38.922265 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.925290 (+ 3025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.927334 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:38.928346 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:38.928527 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:38.933386 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:38.933396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:38.933411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:38.933462 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:38.938022 (+ 4560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:38.938909 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:38.938916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:38.938923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:38.939153 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:38.939639 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:38.939643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:38.939646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:38.940270 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:38.940464 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:38.941000 (+ 536us) server_negotiation.cc:299] Negotiation successful
0621 02:54:38.941394 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12255,"thread_start_us":12453,"threads_started":1}
W20260621 02:54:38.944103 28383 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
I20260621 02:54:38.944253 28383 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1d2b966bb224840a189bda9fbfeaa0c; no voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9
I20260621 02:54:38.944669 28765 raft_consensus.cc:2726] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:54:39.346946 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:39.346134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:36911)
0621 02:54:39.346573 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:39.346706 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260621 02:54:39.464632 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:39.463900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:35905)
0621 02:54:39.464386 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:39.464468 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":262,"threads_started":1}
I20260621 02:54:39.921037 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:39.920361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:35261)
0621 02:54:39.920786 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:39.920862 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":237,"threads_started":1}
I20260621 02:54:40.348430 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:40.347760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:37587)
0621 02:54:40.348195 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:40.348272 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":237,"threads_started":1}
I20260621 02:54:40.466168 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:40.465463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:49587)
0621 02:54:40.465935 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:40.466012 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":262,"threads_started":1}
I20260621 02:54:40.922564 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:40.921891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:41007)
0621 02:54:40.922315 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:40.922400 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260621 02:54:41.144976 28776 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:41.145176 28776 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:41.145856 28776 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
W20260621 02:54:41.146791 28385 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
W20260621 02:54:41.147014 28383 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
I20260621 02:54:41.147142 28383 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1d2b966bb224840a189bda9fbfeaa0c; no voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9
I20260621 02:54:41.147382 28776 raft_consensus.cc:2726] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:54:41.349799 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:41.349157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:53019)
0621 02:54:41.349584 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:41.349657 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":187,"threads_started":1}
I20260621 02:54:41.467643 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:41.466895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:36143)
0621 02:54:41.467362 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:41.467447 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":228,"threads_started":1}
I20260621 02:54:41.924252 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:41.923536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:47749)
0621 02:54:41.924002 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:41.924084 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":245,"threads_started":1}
I20260621 02:54:42.351178 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:42.350567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:47063)
0621 02:54:42.350938 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:42.351018 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":184,"threads_started":1}
I20260621 02:54:42.469314 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:42.468581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:50797)
0621 02:54:42.468981 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:42.469074 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":201,"threads_started":1}
I20260621 02:54:42.925933 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:42.925127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:40463)
0621 02:54:42.925613 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:42.925725 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":274,"threads_started":1}
I20260621 02:54:43.352725 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:43.352022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:44397)
0621 02:54:43.352480 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:43.352564 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260621 02:54:43.470814 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:43.470084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:52715)
0621 02:54:43.470558 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:43.470652 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":294,"threads_started":1}
I20260621 02:54:43.927421 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:43.926760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:33905)
0621 02:54:43.927167 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:43.927245 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
I20260621 02:54:43.964650 28786 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:43.964882 28786 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:43.965610 28786 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
W20260621 02:54:43.966478 28385 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
W20260621 02:54:43.966634 28383 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
I20260621 02:54:43.966773 28383 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1d2b966bb224840a189bda9fbfeaa0c; no voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9
I20260621 02:54:43.967026 28786 raft_consensus.cc:2726] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:54:44.354211 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:44.353571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:37983)
0621 02:54:44.353977 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:44.354051 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260621 02:54:44.472292 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:44.471681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:56701)
0621 02:54:44.472045 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:44.472119 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":192,"threads_started":1}
I20260621 02:54:44.929147 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:44.928396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:33093)
0621 02:54:44.928875 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:44.928963 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":226,"threads_started":1}
I20260621 02:54:45.355708 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:45.354999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:45769)
0621 02:54:45.355463 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:45.355545 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":219,"threads_started":1}
I20260621 02:54:45.473819 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:45.473119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:52373)
0621 02:54:45.473571 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:45.473659 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":255,"threads_started":1}
I20260621 02:54:45.930751 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:45.930073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:40005)
0621 02:54:45.930496 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:45.930574 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":200,"threads_started":1}
I20260621 02:54:46.357364 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:46.356663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:34253)
0621 02:54:46.357075 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:46.357155 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260621 02:54:46.475421 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:46.474730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:36637)
0621 02:54:46.475174 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:46.475255 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260621 02:54:46.678030 28795 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:46.678287 28795 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:46.678987 28795 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
W20260621 02:54:46.679915 28383 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
W20260621 02:54:46.680125 28385 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
I20260621 02:54:46.680294 28385 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1d2b966bb224840a189bda9fbfeaa0c; no voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9
I20260621 02:54:46.680588 28795 raft_consensus.cc:2726] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:54:46.932374 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:46.931676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:44673)
0621 02:54:46.932135 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:46.932216 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":223,"threads_started":1}
I20260621 02:54:47.358875 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:47.358224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:55935)
0621 02:54:47.358657 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:47.358738 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":204,"threads_started":1}
I20260621 02:54:47.477025 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:47.476342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:60727)
0621 02:54:47.476792 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:47.476868 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
I20260621 02:54:47.934051 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:47.933387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:39879)
0621 02:54:47.933783 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:47.933863 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":208,"threads_started":1}
I20260621 02:54:48.360361 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:48.359682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:35203)
0621 02:54:48.360102 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:48.360184 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
I20260621 02:54:48.478504 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:48.477840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:37847)
0621 02:54:48.478265 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:48.478343 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":207,"threads_started":1}
I20260621 02:54:48.935585 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:48.934879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:38447)
0621 02:54:48.935334 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:48.935414 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260621 02:54:49.361948 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:49.361280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:52317)
0621 02:54:49.361717 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:49.361797 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":205,"threads_started":1}
I20260621 02:54:49.480024 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:49.479356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:34197)
0621 02:54:49.479772 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:49.479852 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":209,"threads_started":1}
I20260621 02:54:49.937110 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:49.936411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:51271)
0621 02:54:49.936850 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:49.936949 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260621 02:54:50.327252 28806 raft_consensus.cc:482] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:50.327419 28806 raft_consensus.cc:504] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dedcef199ac6465188f4007f4a6adad9" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 44627 } } peers { permanent_uuid: "387dc238c3da47ff8a571588d1b5fb67" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 43967 } } peers { permanent_uuid: "f1d2b966bb224840a189bda9fbfeaa0c" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 34511 } }
I20260621 02:54:50.328038 28806 leader_election.cc:288] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627), 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967)
W20260621 02:54:50.328987 28383 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 387dc238c3da47ff8a571588d1b5fb67 (127.11.109.3:43967): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
W20260621 02:54:50.329216 28385 leader_election.cc:341] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dedcef199ac6465188f4007f4a6adad9 (127.11.109.2:44627): Not found: Tablet not found: 4ef052f363b044c8961234aa187c9169
I20260621 02:54:50.329380 28385 leader_election.cc:302] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1d2b966bb224840a189bda9fbfeaa0c; no voters: 387dc238c3da47ff8a571588d1b5fb67, dedcef199ac6465188f4007f4a6adad9
I20260621 02:54:50.329641 28806 raft_consensus.cc:2726] T 4ef052f363b044c8961234aa187c9169 P f1d2b966bb224840a189bda9fbfeaa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:54:50.363363 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:50.362744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:42935)
0621 02:54:50.363137 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:50.363211 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20260621 02:54:50.481678 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:50.480837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:51339)
0621 02:54:50.481367 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:50.481459 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":236,"threads_started":1}
I20260621 02:54:50.938874 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:50.938044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:54313)
0621 02:54:50.938604 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:50.938719 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":294,"threads_started":1}
I20260621 02:54:51.364988 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:51.364221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:59769)
0621 02:54:51.364667 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:51.364825 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":200,"threads_started":1}
I20260621 02:54:51.483456 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:51.482706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:59117)
0621 02:54:51.483193 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:51.483272 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":247,"threads_started":1}
I20260621 02:54:51.940618 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:51.939901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:44157)
0621 02:54:51.940344 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:51.940424 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260621 02:54:52.366519 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:52.365835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:52213)
0621 02:54:52.366258 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:52.366332 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260621 02:54:52.485371 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:52.484609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.1:52851)
0621 02:54:52.485025 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:52.485123 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":225,"threads_started":1}
I20260621 02:54:52.942509 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:52.941771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.2:50463)
0621 02:54:52.942250 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:52.942358 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
I20260621 02:54:53.345148 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 28327
I20260621 02:54:53.359547 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:53.346174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.11.109.1:34511)
0621 02:54:53.346764 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:53.346775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:53.346810 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:54:53.346884 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:53.346890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:53.347230 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:53.347625 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:53.348880 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:53.349889 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:53.351069 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:53.351656 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:53.351757 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:53.354362 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:53.354372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:53.354385 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:53.354414 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:53.357062 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:53.357743 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:53.357749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:53.357752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:53.357872 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:53.358196 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:53.358199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:53.358200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:53.358517 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:53.358734 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:53.358974 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 02:54:53.359171 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":295,"threads_started":1}
I20260621 02:54:53.360704 28492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:53.368188 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:53.367528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35281 (local address 127.11.109.3:56313)
0621 02:54:53.367949 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:53.368050 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.62:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":219,"threads_started":1}
I20260621 02:54:53.448926 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 28327
I20260621 02:54:53.466624 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 28501
I20260621 02:54:53.479523 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:53.467406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.11.109.2:44627)
0621 02:54:53.467830 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:53.467835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:53.467850 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:54:53.467901 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:53.467905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:53.467961 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:53.468048 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:53.469420 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:53.470125 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:53.471231 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:53.471630 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:53.471722 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:53.474278 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:53.474282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:53.474285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:53.474313 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:53.476745 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:53.477536 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:53.477544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:53.477549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:53.477698 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:53.478094 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:53.478099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:53.478102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:53.478356 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:53.478632 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:53.478932 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 02:54:53.479139 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":228,"threads_started":1}
I20260621 02:54:53.480145 28624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:53.561720 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 28501
I20260621 02:54:53.578740 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 28632
I20260621 02:54:53.591635 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:53.579533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.11.109.3:43967)
0621 02:54:53.580072 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:53.580077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:53.580094 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:54:53.580150 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:53.580153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:53.580223 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:53.580314 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:53.581737 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:53.582531 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:53.583849 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:53.584264 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:53.584361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:53.586960 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:53.586964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:53.586968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:53.586996 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:53.589519 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:53.590110 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:53.590113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:53.590116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:53.590257 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:53.590551 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:53.590553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:53.590555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:53.590783 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:53.590969 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:53.591201 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:54:53.591350 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":269,"threads_started":1}
I20260621 02:54:53.592330 28755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:53.673422 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 28632
2026-06-21T02:54:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64040 ms)
[----------] 30 tests from SecurityITest (288721 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:54:53 dist-test-slave-bc52 krb5kdc[28825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:54:53 dist-test-slave-bc52 krb5kdc[28825](info): set up 2 sockets
Jun 21 02:54:53 dist-test-slave-bc52 krb5kdc[28825](info): commencing operation
krb5kdc: starting...
W20260621 02:54:54.829259 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:54:54 dist-test-slave-bc52 krb5kdc[28825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:54:55.062673 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37537
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:37537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:54:55.385520 28838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:55.385782 28838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:55.385955 28838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:55.392942 28838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:54:55.393035 28838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:55.393100 28838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:55.393193 28838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:54:55.393299 28838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:54:55.402522 28838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:37537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:55.403820 28838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:55.405568 28838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:55.417856 28843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:55.417798 28844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:55.417881 28846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:55.425369 28838 server_base.cc:1034] running on GCE node
I20260621 02:54:55.426198 28838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:55.427645 28838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:55.428892 28838 hybrid_clock.cc:648] HybridClock initialized: now 1782010495428859 us; error 53 us; skew 500 ppm
I20260621 02:54:55.429322 28838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:55 dist-test-slave-bc52 krb5kdc[28825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010495, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:55.439218 28838 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:54:55.440891 28838 webserver.cc:460] Webserver started at http://127.11.109.62:42075/ using document root <none> and password file <none>
I20260621 02:54:55.441701 28838 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:55.441857 28838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:55.442183 28838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:55.445247 28838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "ea3eaa9e0f8c4f5c93da881acd033901"
format_stamp: "Formatted at 2026-06-21 02:54:55 on dist-test-slave-bc52"
server_key: "b85beef0c12370d34de31e25a50a9d62"
server_key_iv: "988d3534f5e11f2b7ce6b0b89c545102"
server_key_version: "encryptionkey@0"
I20260621 02:54:55.446002 28838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "ea3eaa9e0f8c4f5c93da881acd033901"
format_stamp: "Formatted at 2026-06-21 02:54:55 on dist-test-slave-bc52"
server_key: "b85beef0c12370d34de31e25a50a9d62"
server_key_iv: "988d3534f5e11f2b7ce6b0b89c545102"
server_key_version: "encryptionkey@0"
I20260621 02:54:55.450949 28838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:54:55.454245 28853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:55.455502 28838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:54:55.455683 28838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "ea3eaa9e0f8c4f5c93da881acd033901"
format_stamp: "Formatted at 2026-06-21 02:54:55 on dist-test-slave-bc52"
server_key: "b85beef0c12370d34de31e25a50a9d62"
server_key_iv: "988d3534f5e11f2b7ce6b0b89c545102"
server_key_version: "encryptionkey@0"
I20260621 02:54:55.455868 28838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:55.480661 28838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:55.484846 28838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:55.485150 28838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:55.503871 28838 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:37537
I20260621 02:54:55.503891 28904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:37537 every 8 connection(s)
I20260621 02:54:55.505336 28838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:54:55.515458 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28838
I20260621 02:54:55.515693 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:54:55.516350 11700 external_mini_cluster.cc:1428] Setting key 9271c4daeb095af967c9340f8f20b748
I20260621 02:54:55.516750 28905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:55.531026 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901: Bootstrap starting.
Jun 21 02:54:55 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:55.534648 28905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:55.535715 28905 log.cc:824] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:55.538477 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901: No bootstrap required, opened a new log
I20260621 02:54:55.540330 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:55.518363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.11.109.62:37537)
0621 02:54:55.519221 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:55.519236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:55.519282 (+ 46us) server_negotiation.cc:407] Connection header received
0621 02:54:55.521783 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:55.521811 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:55.522335 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:55.522763 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:55.524078 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:55.526124 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:55.527449 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:55.528550 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:55.528734 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:55.533117 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:55.533125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:55.533138 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:55.533225 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:55.536350 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:55.537082 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:55.537088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:55.537094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:55.537319 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:55.537700 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:55.537707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:55.537713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:55.538176 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:55.538482 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:55.538978 (+ 496us) server_negotiation.cc:299] Negotiation successful
0621 02:54:55.539381 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":357,"threads_started":1}
I20260621 02:54:55.546733 28905 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37537 } }
I20260621 02:54:55.547096 28905 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:55.547210 28905 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3eaa9e0f8c4f5c93da881acd033901, State: Initialized, Role: FOLLOWER
I20260621 02:54:55.547803 28905 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37537 } }
I20260621 02:54:55.548045 28905 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:54:55.548192 28905 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:54:55.548372 28905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:55.550760 28905 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37537 } }
I20260621 02:54:55.551261 28905 leader_election.cc:302] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3eaa9e0f8c4f5c93da881acd033901; no voters:
I20260621 02:54:55.551880 28905 leader_election.cc:288] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:54:55.552006 28910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:54:55.552682 28910 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [term 1 LEADER]: Becoming Leader. State: Replica: ea3eaa9e0f8c4f5c93da881acd033901, State: Running, Role: LEADER
I20260621 02:54:55.553368 28910 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37537 } }
I20260621 02:54:55.554147 28905 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:54:55.556324 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3eaa9e0f8c4f5c93da881acd033901. Latest consensus state: current_term: 1 leader_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37537 } } }
I20260621 02:54:55.556756 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:55.556634 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3eaa9e0f8c4f5c93da881acd033901" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37537 } } }
I20260621 02:54:55.556927 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901 [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:55.558163 28919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:54:55.562937 28919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:54:55.571864 28919 catalog_manager.cc:1347] Generated new cluster ID: fdce6a1c0e9048bfaa1f64d53d55f5cd
I20260621 02:54:55.572008 28919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:54:55.581666 28919 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:54:55.583482 28919 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:54:55.594143 28919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901: Generated new TSK 0
I20260621 02:54:55.594902 28919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:54:55.648304 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37537
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:54:55.975317 28931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:55.975584 28931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:55.975759 28931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:55.982882 28931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:55.982978 28931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:55.983194 28931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:54:55.992802 28931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:55.993971 28931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:55.995283 28931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:56.008428 28936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:56.008345 28939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:56.008396 28937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:56.009012 28931 server_base.cc:1034] running on GCE node
I20260621 02:54:56.009691 28931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:56.010681 28931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:56.011915 28931 hybrid_clock.cc:648] HybridClock initialized: now 1782010496011855 us; error 55 us; skew 500 ppm
I20260621 02:54:56.012226 28931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:56 dist-test-slave-bc52 krb5kdc[28825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:56.015331 28931 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:54:56.016798 28931 webserver.cc:460] Webserver started at http://127.11.109.1:32965/ using document root <none> and password file <none>
I20260621 02:54:56.017562 28931 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:56.017711 28931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:56.018016 28931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:56.021003 28931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7"
format_stamp: "Formatted at 2026-06-21 02:54:56 on dist-test-slave-bc52"
server_key: "e382dc291c2bb568465185776499edda"
server_key_iv: "6676f369ea95fe0eb1c76a6403f9f9bd"
server_key_version: "encryptionkey@0"
I20260621 02:54:56.021826 28931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7"
format_stamp: "Formatted at 2026-06-21 02:54:56 on dist-test-slave-bc52"
server_key: "e382dc291c2bb568465185776499edda"
server_key_iv: "6676f369ea95fe0eb1c76a6403f9f9bd"
server_key_version: "encryptionkey@0"
I20260621 02:54:56.026786 28931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:54:56.030210 28946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:56.031414 28931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:54:56.031594 28931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7"
format_stamp: "Formatted at 2026-06-21 02:54:56 on dist-test-slave-bc52"
server_key: "e382dc291c2bb568465185776499edda"
server_key_iv: "6676f369ea95fe0eb1c76a6403f9f9bd"
server_key_version: "encryptionkey@0"
I20260621 02:54:56.031805 28931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:56.053680 28931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:56.058151 28931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:56.058449 28931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:56.059422 28931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:56.060917 28931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:56.061028 28931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:56.061226 28931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:56.061295 28931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260621 02:54:56.094369 28931 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:39283
I20260621 02:54:56.094478 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:39283 every 8 connection(s)
I20260621 02:54:56.095665 28931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:54:56.101927 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 28931
I20260621 02:54:56.102154 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:54:56.102835 11700 external_mini_cluster.cc:1428] Setting key c9a8f60336019f426c7baf5d4eb3c7f0
Jun 21 02:54:56 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:56.118131 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:56.098440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:46835 (local address 127.11.109.62:37537)
0621 02:54:56.098890 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:56.098895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:56.099936 (+ 1041us) server_negotiation.cc:407] Connection header received
0621 02:54:56.101793 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:56.101799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:56.101901 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:56.102028 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:56.103742 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.104609 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.106197 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.106633 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.106766 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:56.111204 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:56.111213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:56.111220 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:56.111268 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:56.114820 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:56.115784 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:56.115787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:56.115789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:56.115902 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:56.116383 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:56.116386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:56.116387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:56.116633 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:56.116748 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:56.117686 (+ 938us) server_negotiation.cc:299] Negotiation successful
0621 02:54:56.117870 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260621 02:54:56.119637 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:56.099050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37537 (local address 127.11.109.1:46835)
0621 02:54:56.099758 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:56.099800 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:56.101503 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:56.102330 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:56.102338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:56.102926 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:56.103522 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:56.103572 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.104753 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.104757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:56.106043 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:56.106055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.106886 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.106895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:56.107224 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:56.108006 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:56.108041 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:56.110970 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:56.114969 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:56.114981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:56.115001 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:56.115657 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:56.116012 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:56.116019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:56.116026 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:56.116270 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:56.116770 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:56.116781 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:56.117288 (+ 507us) client_negotiation.cc:769] Sending connection context
0621 02:54:56.117583 (+ 295us) client_negotiation.cc:240] Negotiation successful
0621 02:54:56.118174 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":266,"threads_started":1}
I20260621 02:54:56.121371 29059 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37537
I20260621 02:54:56.121717 29059 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:56.122593 29059 heartbeater.cc:511] Master 127.11.109.62:37537 requested a full tablet report, sending...
I20260621 02:54:56.124796 28870 ts_manager.cc:194] Registered new tserver with Master: 4d3e67cdc52b4eb69c9e09a87c437ed7 (127.11.109.1:39283)
I20260621 02:54:56.126341 28870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:46835
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:54:56.161947 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37537
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:54:56.480289 29065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:56.480574 29065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:56.480777 29065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:56.488090 29065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:56.488198 29065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:56.488382 29065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:54:56.498423 29065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:56.499747 29065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:56.501071 29065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:56.513931 29070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:56.513972 29071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:56.514741 29073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:56.514935 29065 server_base.cc:1034] running on GCE node
I20260621 02:54:56.515606 29065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:56.516776 29065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:56.518008 29065 hybrid_clock.cc:648] HybridClock initialized: now 1782010496517942 us; error 57 us; skew 500 ppm
I20260621 02:54:56.518357 29065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:56 dist-test-slave-bc52 krb5kdc[28825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:56.522192 29065 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:54:56.523622 29065 webserver.cc:460] Webserver started at http://127.11.109.2:33215/ using document root <none> and password file <none>
I20260621 02:54:56.524257 29065 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:56.524355 29065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:56.524633 29065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:56.527539 29065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "df8a415a97e947e292a7b4c67aae16b6"
format_stamp: "Formatted at 2026-06-21 02:54:56 on dist-test-slave-bc52"
server_key: "857ce2a7872101206a20105b1b927026"
server_key_iv: "ae0933c3b0f53df6eb80617e79f00fc5"
server_key_version: "encryptionkey@0"
I20260621 02:54:56.528223 29065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "df8a415a97e947e292a7b4c67aae16b6"
format_stamp: "Formatted at 2026-06-21 02:54:56 on dist-test-slave-bc52"
server_key: "857ce2a7872101206a20105b1b927026"
server_key_iv: "ae0933c3b0f53df6eb80617e79f00fc5"
server_key_version: "encryptionkey@0"
I20260621 02:54:56.533340 29065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 02:54:56.536782 29080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:56.538355 29065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:56.538488 29065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "df8a415a97e947e292a7b4c67aae16b6"
format_stamp: "Formatted at 2026-06-21 02:54:56 on dist-test-slave-bc52"
server_key: "857ce2a7872101206a20105b1b927026"
server_key_iv: "ae0933c3b0f53df6eb80617e79f00fc5"
server_key_version: "encryptionkey@0"
I20260621 02:54:56.538654 29065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:56.565708 29065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:56.569628 29065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:56.569887 29065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:56.570778 29065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:56.572046 29065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:56.572119 29065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:56.572201 29065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:56.572244 29065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:56.604959 29065 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:38049
I20260621 02:54:56.605006 29192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:38049 every 8 connection(s)
I20260621 02:54:56.606238 29065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:54:56.614380 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29065
I20260621 02:54:56.614565 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:54:56.614989 11700 external_mini_cluster.cc:1428] Setting key af56c88dad0b2b0a400a3a7131b85a0c
Jun 21 02:54:56 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010496, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:56.629347 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:56.608724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:57587 (local address 127.11.109.62:37537)
0621 02:54:56.608946 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:56.608952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:56.610132 (+ 1180us) server_negotiation.cc:407] Connection header received
0621 02:54:56.611856 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:56.611860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:56.611927 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:56.612012 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:56.615615 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.616557 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.618375 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.619156 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.619295 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:56.623010 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:56.623018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:56.623026 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:56.623070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:56.626122 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:56.627043 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:56.627047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:56.627049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:56.627165 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:56.627629 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:56.627632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:56.627633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:56.627889 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:56.627994 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:56.628852 (+ 858us) server_negotiation.cc:299] Negotiation successful
0621 02:54:56.629029 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:54:56.630841 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:56.609300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37537 (local address 127.11.109.2:57587)
0621 02:54:56.609975 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:56.610013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:56.611605 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:56.612231 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:56.612238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:56.612840 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:56.613869 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:56.613893 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.616703 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.616707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:56.618223 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:56.618235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:56.619348 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:56.619353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:56.619546 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:56.620310 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:56.620345 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:56.622759 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:56.626273 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:56.626285 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:56.626304 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:56.626918 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:56.627270 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:56.627277 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:56.627283 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:56.627518 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:56.628012 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:56.628022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:56.628458 (+ 436us) client_negotiation.cc:769] Sending connection context
0621 02:54:56.628766 (+ 308us) client_negotiation.cc:240] Negotiation successful
0621 02:54:56.629374 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":259,"threads_started":1}
I20260621 02:54:56.632323 29193 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37537
I20260621 02:54:56.632635 29193 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:56.633416 29193 heartbeater.cc:511] Master 127.11.109.62:37537 requested a full tablet report, sending...
I20260621 02:54:56.634873 28870 ts_manager.cc:194] Registered new tserver with Master: df8a415a97e947e292a7b4c67aae16b6 (127.11.109.2:38049)
I20260621 02:54:56.635706 28870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:57587
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:54:56.673848 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37537
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:54:56.993132 29198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:56.993516 29198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:56.993698 29198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:57.000877 29198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:57.001027 29198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:57.001240 29198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:54:57.010782 29198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:57.012125 29198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:57.013557 29198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:57.026660 29206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:57.026762 29204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:57.026698 29203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:57.027546 29198 server_base.cc:1034] running on GCE node
I20260621 02:54:57.028218 29198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:57.029150 29198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:57.030416 29198 hybrid_clock.cc:648] HybridClock initialized: now 1782010497030373 us; error 35 us; skew 500 ppm
I20260621 02:54:57.030755 29198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:54:57 dist-test-slave-bc52 krb5kdc[28825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:54:57.033951 29198 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:54:57.035528 29198 webserver.cc:460] Webserver started at http://127.11.109.3:33455/ using document root <none> and password file <none>
I20260621 02:54:57.036211 29198 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:57.036365 29198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:57.036700 29198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:57.039623 29198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "bb8f71f7960d4462896a12b1966f54a8"
format_stamp: "Formatted at 2026-06-21 02:54:57 on dist-test-slave-bc52"
server_key: "bace3811531af7838c419a0d85e30954"
server_key_iv: "7998c36b40c8a2d960541509fa625f4f"
server_key_version: "encryptionkey@0"
I20260621 02:54:57.040416 29198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "bb8f71f7960d4462896a12b1966f54a8"
format_stamp: "Formatted at 2026-06-21 02:54:57 on dist-test-slave-bc52"
server_key: "bace3811531af7838c419a0d85e30954"
server_key_iv: "7998c36b40c8a2d960541509fa625f4f"
server_key_version: "encryptionkey@0"
I20260621 02:54:57.045687 29198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:54:57.048924 29213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:57.050313 29198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:57.050494 29198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "bb8f71f7960d4462896a12b1966f54a8"
format_stamp: "Formatted at 2026-06-21 02:54:57 on dist-test-slave-bc52"
server_key: "bace3811531af7838c419a0d85e30954"
server_key_iv: "7998c36b40c8a2d960541509fa625f4f"
server_key_version: "encryptionkey@0"
I20260621 02:54:57.050706 29198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:57.068842 29198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:57.073201 29198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:57.073503 29198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:57.074440 29198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:57.075749 29198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:57.075858 29198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:57.076042 29198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:57.076153 29198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:57.109607 29198 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:32841
I20260621 02:54:57.109643 29325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:32841 every 8 connection(s)
I20260621 02:54:57.110824 29198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:54:57.117249 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29198
I20260621 02:54:57.117430 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:54:57.117868 11700 external_mini_cluster.cc:1428] Setting key 90e4123b7930dda9a66bb027afc9237e
Jun 21 02:54:57 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010497, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:54:57.129982 29059 heartbeater.cc:503] Master 127.11.109.62:37537 was elected leader, sending a full tablet report...
I20260621 02:54:57.130859 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.113334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:55857 (local address 127.11.109.62:37537)
0621 02:54:57.113508 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.113512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.114666 (+ 1154us) server_negotiation.cc:407] Connection header received
0621 02:54:57.116640 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.116646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.116754 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.116887 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:57.118670 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.119631 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.121652 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.122044 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.122152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.125303 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:57.125307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:57.125311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:57.125342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:57.128039 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.128668 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.128671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.128674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.128799 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.129161 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.129164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.129166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.129486 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:57.129604 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.130302 (+ 698us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.130518 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:54:57.132017 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.113822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37537 (local address 127.11.109.3:55857)
0621 02:54:57.114498 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:57.114533 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:57.116380 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:57.117124 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:57.117133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:57.117811 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:57.118465 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.118490 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.119810 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.119814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.121509 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.121520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.122236 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.122241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.122417 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.123016 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:54:57.123040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:57.125061 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:57.128155 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:57.128161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:57.128172 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:57.128541 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:57.128894 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:54:57.128897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:54:57.128899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:54:57.129053 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:54:57.129600 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:57.129605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:57.129902 (+ 297us) client_negotiation.cc:769] Sending connection context
0621 02:54:57.130172 (+ 270us) client_negotiation.cc:240] Negotiation successful
0621 02:54:57.130636 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260621 02:54:57.133592 29326 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37537
I20260621 02:54:57.133905 29326 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:57.134608 29326 heartbeater.cc:511] Master 127.11.109.62:37537 requested a full tablet report, sending...
I20260621 02:54:57.135954 28869 ts_manager.cc:194] Registered new tserver with Master: bb8f71f7960d4462896a12b1966f54a8 (127.11.109.3:32841)
I20260621 02:54:57.136794 28869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:55857
I20260621 02:54:57.145350 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:54:57.162339 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.149931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.11.109.62:37537)
0621 02:54:57.150106 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.150110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.150554 (+ 444us) server_negotiation.cc:407] Connection header received
0621 02:54:57.150676 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.150680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.150735 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.150812 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:57.152245 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.152915 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.154334 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.154899 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.155006 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.157395 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:57.157399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:57.157402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:57.157430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:57.159986 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.160670 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.160674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.160676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.160800 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.161121 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.161124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.161126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.161494 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:57.161681 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.161966 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.162118 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:54:57.166488 28869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:54:57.173146 28869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:54:57.194939 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.185082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39283 (local address 127.0.0.1:55064)
0621 02:54:57.187064 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:57.187080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:57.187190 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:57.188063 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:57.188073 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:57.188113 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:57.188542 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.188555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.190668 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.190675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.192050 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.192061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.194000 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.194006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.194215 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.194265 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 02:54:57.194405 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 02:54:57.194594 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"thread_start_us":1701,"threads_started":1}
I20260621 02:54:57.194958 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.185822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.11.109.1:39283)
0621 02:54:57.187305 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.187317 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.187353 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:54:57.187472 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.187482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.187784 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.188767 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:57.188987 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.190532 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.192291 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.193768 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.193900 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.194105 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.194477 (+ 372us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.194638 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247,"thread_start_us":2126,"threads_started":1}
I20260621 02:54:57.196945 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.184845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:38049 (local address 127.0.0.1:59222)
0621 02:54:57.185732 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:57.185789 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:57.185982 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:57.189659 (+ 3677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:57.189664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:57.189692 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:57.190138 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.190153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.192173 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.192181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.194308 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.194319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.196112 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.196118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.196266 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.196310 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 02:54:57.196411 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:54:57.196537 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":1,"thread_start_us":395,"threads_started":1}
I20260621 02:54:57.197297 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.187688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.11.109.3:32841)
0621 02:54:57.189025 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.189036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.189068 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:54:57.189156 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.189164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.189414 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.189585 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:57.190635 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.192375 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.194641 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.195994 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.196138 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.196363 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.196753 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.196911 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":993,"threads_started":1}
I20260621 02:54:57.197567 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.184966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.11.109.2:38049)
0621 02:54:57.189034 (+ 4068us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.189042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.189067 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:54:57.189165 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.189202 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.189446 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.189997 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:57.190452 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.192025 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.194513 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.195895 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.196068 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.196263 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.197043 (+ 780us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.197239 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3550,"thread_start_us":3391,"threads_started":1}
I20260621 02:54:57.197989 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.187126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:32841 (local address 127.0.0.1:39326)
0621 02:54:57.187627 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:57.187645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:57.187743 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:57.189839 (+ 2096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:57.189848 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:57.189871 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:57.190453 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.190467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.192530 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.192536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.194488 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.194500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.196414 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.196421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.196570 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.196600 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:54:57.196693 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:54:57.196804 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":30,"thread_start_us":706,"threads_started":1}
I20260621 02:54:57.199142 28994 tablet_service.cc:1467] Processing CreateTablet for tablet 9caba65e27bb44148809e1e67c32af37 (DEFAULT_TABLE table=test-table [id=ee29e0454c524e2fbc1397bf13471a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:57.200734 28994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caba65e27bb44148809e1e67c32af37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:57.201092 29261 tablet_service.cc:1467] Processing CreateTablet for tablet 9caba65e27bb44148809e1e67c32af37 (DEFAULT_TABLE table=test-table [id=ee29e0454c524e2fbc1397bf13471a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:57.201050 29128 tablet_service.cc:1467] Processing CreateTablet for tablet 9caba65e27bb44148809e1e67c32af37 (DEFAULT_TABLE table=test-table [id=ee29e0454c524e2fbc1397bf13471a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:57.202262 29261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caba65e27bb44148809e1e67c32af37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:57.202355 29128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9caba65e27bb44148809e1e67c32af37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:57.213091 29344 tablet_bootstrap.cc:492] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Bootstrap starting.
I20260621 02:54:57.216099 29346 tablet_bootstrap.cc:492] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Bootstrap starting.
I20260621 02:54:57.216539 29344 tablet_bootstrap.cc:654] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:57.218160 29344 log.cc:824] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:57.218685 29345 tablet_bootstrap.cc:492] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Bootstrap starting.
I20260621 02:54:57.219961 29346 tablet_bootstrap.cc:654] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:57.221251 29346 log.cc:824] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:57.221552 29345 tablet_bootstrap.cc:654] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:57.222410 29344 tablet_bootstrap.cc:492] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: No bootstrap required, opened a new log
I20260621 02:54:57.222699 29344 ts_tablet_manager.cc:1397] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260621 02:54:57.222745 29345 log.cc:824] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:57.224244 29346 tablet_bootstrap.cc:492] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: No bootstrap required, opened a new log
I20260621 02:54:57.224488 29346 ts_tablet_manager.cc:1397] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 02:54:57.225346 29345 tablet_bootstrap.cc:492] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: No bootstrap required, opened a new log
I20260621 02:54:57.225641 29345 ts_tablet_manager.cc:1397] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260621 02:54:57.232367 29345 raft_consensus.cc:348] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.232751 29345 raft_consensus.cc:374] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:57.232987 29345 raft_consensus.cc:729] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3e67cdc52b4eb69c9e09a87c437ed7, State: Initialized, Role: FOLLOWER
I20260621 02:54:57.233400 29344 raft_consensus.cc:348] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.233837 29344 raft_consensus.cc:374] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:57.233711 29345 consensus_queue.cc:260] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.233935 29344 raft_consensus.cc:729] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb8f71f7960d4462896a12b1966f54a8, State: Initialized, Role: FOLLOWER
I20260621 02:54:57.234652 29344 consensus_queue.cc:260] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.235051 29346 raft_consensus.cc:348] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.235512 29345 ts_tablet_manager.cc:1428] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260621 02:54:57.235728 29346 raft_consensus.cc:374] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:57.235935 29346 raft_consensus.cc:729] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8a415a97e947e292a7b4c67aae16b6, State: Initialized, Role: FOLLOWER
I20260621 02:54:57.236756 29346 consensus_queue.cc:260] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.239737 29326 heartbeater.cc:503] Master 127.11.109.62:37537 was elected leader, sending a full tablet report...
I20260621 02:54:57.240517 29344 ts_tablet_manager.cc:1428] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260621 02:54:57.241530 29193 heartbeater.cc:503] Master 127.11.109.62:37537 was elected leader, sending a full tablet report...
I20260621 02:54:57.241925 29346 ts_tablet_manager.cc:1428] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260621 02:54:57.318506 29352 raft_consensus.cc:482] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:54:57.318768 29352 raft_consensus.cc:504] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.320407 29352 leader_election.cc:288] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3e67cdc52b4eb69c9e09a87c437ed7 (127.11.109.1:39283), bb8f71f7960d4462896a12b1966f54a8 (127.11.109.3:32841)
I20260621 02:54:57.326776 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.320865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:43043 (local address 127.11.109.1:39283)
0621 02:54:57.321122 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.321128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.321462 (+ 334us) server_negotiation.cc:407] Connection header received
0621 02:54:57.321880 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.321884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.321943 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.322053 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:57.322587 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.323319 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.325045 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.325864 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.326049 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.326130 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.326404 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.326524 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 02:54:57.326860 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.320588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39283 (local address 127.11.109.2:43043)
0621 02:54:57.321357 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:57.321385 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:57.321568 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:57.322078 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:57.322083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:57.322109 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:57.322440 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.322449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.323453 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.323461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.324886 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.324898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.326074 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.326081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.326246 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.326269 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:54:57.326379 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:54:57.326511 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":368,"threads_started":1}
I20260621 02:54:57.327176 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.321569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:58801 (local address 127.11.109.3:32841)
0621 02:54:57.321796 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.321800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.322181 (+ 381us) server_negotiation.cc:407] Connection header received
0621 02:54:57.322414 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.322419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.322484 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.322584 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:57.323084 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.323784 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.325524 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.326273 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.326465 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.326535 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.326819 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.326938 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:54:57.327245 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.321358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:32841 (local address 127.11.109.2:58801)
0621 02:54:57.322077 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:57.322091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:57.322267 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:57.322596 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:57.322600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:57.322621 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:57.322943 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.322950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.323921 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.323926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.325383 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:57.325394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.326485 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.326491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:57.326665 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.326687 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:54:57.326793 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:54:57.326919 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":471,"threads_started":1}
I20260621 02:54:57.328059 29014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caba65e27bb44148809e1e67c32af37" candidate_uuid: "df8a415a97e947e292a7b4c67aae16b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" is_pre_election: true
I20260621 02:54:57.328411 29281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caba65e27bb44148809e1e67c32af37" candidate_uuid: "df8a415a97e947e292a7b4c67aae16b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8f71f7960d4462896a12b1966f54a8" is_pre_election: true
I20260621 02:54:57.328667 29014 raft_consensus.cc:2454] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8a415a97e947e292a7b4c67aae16b6 in term 0.
I20260621 02:54:57.328930 29281 raft_consensus.cc:2454] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8a415a97e947e292a7b4c67aae16b6 in term 0.
I20260621 02:54:57.329437 29084 leader_election.cc:302] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3e67cdc52b4eb69c9e09a87c437ed7, df8a415a97e947e292a7b4c67aae16b6; no voters:
I20260621 02:54:57.329792 29352 raft_consensus.cc:2781] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:54:57.329911 29352 raft_consensus.cc:482] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:54:57.330063 29352 raft_consensus.cc:3037] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:57.332201 29352 raft_consensus.cc:504] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.332754 29352 leader_election.cc:288] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3e67cdc52b4eb69c9e09a87c437ed7 (127.11.109.1:39283), bb8f71f7960d4462896a12b1966f54a8 (127.11.109.3:32841)
I20260621 02:54:57.333145 29014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caba65e27bb44148809e1e67c32af37" candidate_uuid: "df8a415a97e947e292a7b4c67aae16b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7"
I20260621 02:54:57.333251 29281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9caba65e27bb44148809e1e67c32af37" candidate_uuid: "df8a415a97e947e292a7b4c67aae16b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb8f71f7960d4462896a12b1966f54a8"
I20260621 02:54:57.333377 29014 raft_consensus.cc:3037] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:57.333467 29281 raft_consensus.cc:3037] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:57.335598 29014 raft_consensus.cc:2454] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8a415a97e947e292a7b4c67aae16b6 in term 1.
I20260621 02:54:57.335916 29281 raft_consensus.cc:2454] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8a415a97e947e292a7b4c67aae16b6 in term 1.
I20260621 02:54:57.335963 29084 leader_election.cc:302] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3e67cdc52b4eb69c9e09a87c437ed7, df8a415a97e947e292a7b4c67aae16b6; no voters:
I20260621 02:54:57.336222 29352 raft_consensus.cc:2781] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:54:57.336679 29352 raft_consensus.cc:686] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 1 LEADER]: Becoming Leader. State: Replica: df8a415a97e947e292a7b4c67aae16b6, State: Running, Role: LEADER
I20260621 02:54:57.337138 29352 consensus_queue.cc:237] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } }
I20260621 02:54:57.341744 28870 catalog_manager.cc:5495] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8a415a97e947e292a7b4c67aae16b6 (127.11.109.2). New cstate: current_term: 1 leader_uuid: "df8a415a97e947e292a7b4c67aae16b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a415a97e947e292a7b4c67aae16b6" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 } health_report { overall_health: UNKNOWN } } }
W20260621 02:54:57.348104 29060 tablet.cc:2369] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:54:57.358096 29194 tablet.cc:2369] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:54:57.362350 29327 tablet.cc:2369] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:54:57.388410 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.383475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.11.109.2:38049)
0621 02:54:57.383656 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.383660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.383681 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:54:57.383876 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.383880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.383952 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.384072 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:54:57.384413 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.385107 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.386543 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.386922 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.387012 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.387485 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:54:57.387942 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:54:57.388045 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.388142 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.388202 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:54:57.396086 29281 raft_consensus.cc:1261] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 1 FOLLOWER]: Refusing update from remote peer df8a415a97e947e292a7b4c67aae16b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:54:57.396178 29014 raft_consensus.cc:1261] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 1 FOLLOWER]: Refusing update from remote peer df8a415a97e947e292a7b4c67aae16b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:54:57.397047 29355 consensus_queue.cc:1035] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8f71f7960d4462896a12b1966f54a8" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:54:57.397464 29352 consensus_queue.cc:1035] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3e67cdc52b4eb69c9e09a87c437ed7" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:54:57.408299 29358 mvcc.cc:205] Tried to move back new op lower bound from 7299114997325295616 to 7299114997096992768. Current Snapshot: MvccSnapshot[applied={T|T < 7299114997325295616}]
I20260621 02:54:57.409766 29359 mvcc.cc:205] Tried to move back new op lower bound from 7299114997325295616 to 7299114997096992768. Current Snapshot: MvccSnapshot[applied={T|T < 7299114997325295616}]
I20260621 02:54:57.421941 28870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:54:57.422258 28870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:54:57.425801 28870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3eaa9e0f8c4f5c93da881acd033901: Sending DeleteTablet for 3 replicas of tablet 9caba65e27bb44148809e1e67c32af37
I20260621 02:54:57.426918 29128 tablet_service.cc:1514] Processing DeleteTablet for tablet 9caba65e27bb44148809e1e67c32af37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:54:57 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:59222
I20260621 02:54:57.427067 28994 tablet_service.cc:1514] Processing DeleteTablet for tablet 9caba65e27bb44148809e1e67c32af37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:54:57 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:55064
I20260621 02:54:57.427585 29261 tablet_service.cc:1514] Processing DeleteTablet for tablet 9caba65e27bb44148809e1e67c32af37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:54:57 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:39326
I20260621 02:54:57.427717 29367 tablet_replica.cc:330] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: stopping tablet replica
I20260621 02:54:57.427953 29368 tablet_replica.cc:330] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: stopping tablet replica
I20260621 02:54:57.428128 29367 raft_consensus.cc:2229] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:54:57.428428 29368 raft_consensus.cc:2229] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:54:57.428889 29367 raft_consensus.cc:2258] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:54:57.428920 29368 raft_consensus.cc:2258] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:54:57.429816 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 28931
I20260621 02:54:57.431042 29367 ts_tablet_manager.cc:1905] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:54:57.435086 29368 ts_tablet_manager.cc:1905] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:54:57.440081 29367 ts_tablet_manager.cc:1918] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:54:57.440351 29367 log.cc:1196] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/9caba65e27bb44148809e1e67c32af37
I20260621 02:54:57.440984 29367 ts_tablet_manager.cc:1939] T 9caba65e27bb44148809e1e67c32af37 P df8a415a97e947e292a7b4c67aae16b6: Deleting consensus metadata
I20260621 02:54:57.441984 29369 tablet_replica.cc:330] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: stopping tablet replica
I20260621 02:54:57.442899 28855 catalog_manager.cc:4841] TS df8a415a97e947e292a7b4c67aae16b6 (127.11.109.2:38049): tablet 9caba65e27bb44148809e1e67c32af37 (table test-table [id=ee29e0454c524e2fbc1397bf13471a22]) successfully deleted
I20260621 02:54:57.442550 29369 raft_consensus.cc:2229] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:54:57.443617 29369 raft_consensus.cc:2258] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:54:57.446128 29368 ts_tablet_manager.cc:1918] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:54:57.446313 29368 log.cc:1196] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/9caba65e27bb44148809e1e67c32af37
I20260621 02:54:57.446913 29368 ts_tablet_manager.cc:1939] T 9caba65e27bb44148809e1e67c32af37 P 4d3e67cdc52b4eb69c9e09a87c437ed7: Deleting consensus metadata
I20260621 02:54:57.448544 28857 catalog_manager.cc:4841] TS 4d3e67cdc52b4eb69c9e09a87c437ed7 (127.11.109.1:39283): tablet 9caba65e27bb44148809e1e67c32af37 (table test-table [id=ee29e0454c524e2fbc1397bf13471a22]) successfully deleted
I20260621 02:54:57.449492 29369 ts_tablet_manager.cc:1905] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:54:57 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:54:57.454970 29369 ts_tablet_manager.cc:1918] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:54:57.455123 29369 log.cc:1196] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/9caba65e27bb44148809e1e67c32af37
I20260621 02:54:57.455652 29369 ts_tablet_manager.cc:1939] T 9caba65e27bb44148809e1e67c32af37 P bb8f71f7960d4462896a12b1966f54a8: Deleting consensus metadata
I20260621 02:54:57.457149 28857 catalog_manager.cc:4841] TS bb8f71f7960d4462896a12b1966f54a8 (127.11.109.3:32841): tablet 9caba65e27bb44148809e1e67c32af37 (table test-table [id=ee29e0454c524e2fbc1397bf13471a22]) successfully deleted
I20260621 02:54:57.462563 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.431028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.11.109.1:39283)
0621 02:54:57.431200 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.431205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.444364 (+ 13159us) server_negotiation.cc:407] Connection header received
0621 02:54:57.444494 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.444501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.444614 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.444736 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:57.446569 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.447716 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.449596 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.450187 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.450323 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.455800 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:57.455815 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:57.455832 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:57.455870 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:57.459652 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.460395 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.460407 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.460412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.460585 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.460947 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.460952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.460955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.461470 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:57.461769 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.461960 (+ 191us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.462187 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:54:57.463272 29054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:57.578428 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 28931
I20260621 02:54:57.595554 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29065
Jun 21 02:54:57 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:54:57.613071 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.596522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.11.109.2:38049)
0621 02:54:57.596751 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.596757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.596774 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:54:57.596838 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.596841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.596913 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.597031 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:57.598344 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.599133 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.600442 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.600879 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.601022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.605682 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:57.605693 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:57.605709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:57.605757 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:57.609998 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.610815 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.610824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.610827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.610999 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.611458 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.611463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.611467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.611837 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:57.612157 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.612463 (+ 306us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.612690 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:54:57.614033 29188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:57.724844 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29065
I20260621 02:54:57.739972 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29198
Jun 21 02:54:57 dist-test-slave-bc52 krb5kdc[28825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:54:57.754088 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:57.740748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.11.109.3:32841)
0621 02:54:57.740977 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:57.740987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:57.741016 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:54:57.741278 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:57.741282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:57.741354 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:57.741436 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:57.742663 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.743351 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.744609 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:57.744987 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:57.745075 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:57.749273 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:57.749285 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:57.749301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:54:57.749332 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:57.751697 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.752279 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.752285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.752287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.752433 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:54:57.752784 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:54:57.752787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:54:57.752789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:54:57.753083 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:57.753495 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:57.753606 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 02:54:57.753753 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 02:54:57.754782 29321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:57.845152 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29198
I20260621 02:54:57.862159 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 28838
I20260621 02:54:57.863291 28900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:54:57.972748 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 28838
2026-06-21T02:54:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4305 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T02:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:54:58Z Disabled control of system clock
I20260621 02:54:58.050361 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37105
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:54:58.369628 29378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:58.369908 29378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:58.370067 29378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:58.377097 29378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:54:58.377209 29378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:58.377300 29378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:58.377409 29378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:54:58.377490 29378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:54:58.386734 29378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:37105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:58.387962 29378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:58.389760 29378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:58.401273 29386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:58.401314 29384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:58.401371 29383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:58.402608 29378 server_base.cc:1034] running on GCE node
I20260621 02:54:58.403368 29378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:58.404655 29378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:58.405856 29378 hybrid_clock.cc:648] HybridClock initialized: now 1782010498405808 us; error 38 us; skew 500 ppm
I20260621 02:54:58.406144 29378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:54:58.408326 29378 webserver.cc:460] Webserver started at http://127.11.109.62:34683/ using document root <none> and password file <none>
I20260621 02:54:58.408936 29378 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:58.409017 29378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:58.409356 29378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:58.412097 29378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "1f3780e55163464e82400e43a4a9018e"
format_stamp: "Formatted at 2026-06-21 02:54:58 on dist-test-slave-bc52"
server_key: "d0ff32af35aef3c843613d478b2b20bc"
server_key_iv: "cd4f3cc65009d1490d88993dbb598dc8"
server_key_version: "encryptionkey@0"
I20260621 02:54:58.412848 29378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "1f3780e55163464e82400e43a4a9018e"
format_stamp: "Formatted at 2026-06-21 02:54:58 on dist-test-slave-bc52"
server_key: "d0ff32af35aef3c843613d478b2b20bc"
server_key_iv: "cd4f3cc65009d1490d88993dbb598dc8"
server_key_version: "encryptionkey@0"
I20260621 02:54:58.417682 29378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260621 02:54:58.420943 29392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:58.422581 29378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:54:58.422778 29378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "1f3780e55163464e82400e43a4a9018e"
format_stamp: "Formatted at 2026-06-21 02:54:58 on dist-test-slave-bc52"
server_key: "d0ff32af35aef3c843613d478b2b20bc"
server_key_iv: "cd4f3cc65009d1490d88993dbb598dc8"
server_key_version: "encryptionkey@0"
I20260621 02:54:58.422983 29378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:58.439363 29378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:58.440151 29378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:58.440446 29378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:58.458596 29378 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:37105
I20260621 02:54:58.458629 29443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:37105 every 8 connection(s)
I20260621 02:54:58.460016 29378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:54:58.463755 29444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:58.469987 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29378
I20260621 02:54:58.470185 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:54:58.470642 11700 external_mini_cluster.cc:1428] Setting key fad518851f84d9e2694b176da1010a96
I20260621 02:54:58.474563 29444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e: Bootstrap starting.
I20260621 02:54:58.477998 29444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:58.478988 29444 log.cc:824] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:58.481796 29444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e: No bootstrap required, opened a new log
I20260621 02:54:58.483979 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:58.472620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.11.109.62:37105)
0621 02:54:58.473614 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:58.473625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:58.473664 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:54:58.475396 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:58.475417 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:58.475845 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:58.476165 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:58.476772 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:58.478146 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:58.479550 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:58.480280 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:58.480484 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:58.481678 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:58.481686 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:58.481699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:54:58.482290 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:58.482394 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:54:58.482415 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:58.482558 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:58.482773 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 02:54:58.483069 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":348,"threads_started":1}
I20260621 02:54:58.489512 29444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3780e55163464e82400e43a4a9018e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37105 } }
I20260621 02:54:58.489886 29444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:58.489997 29444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3780e55163464e82400e43a4a9018e, State: Initialized, Role: FOLLOWER
I20260621 02:54:58.490522 29444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3780e55163464e82400e43a4a9018e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37105 } }
I20260621 02:54:58.490770 29444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:54:58.490918 29444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:54:58.491097 29444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:54:58.493649 29444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3780e55163464e82400e43a4a9018e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37105 } }
I20260621 02:54:58.494127 29444 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f3780e55163464e82400e43a4a9018e; no voters:
I20260621 02:54:58.494671 29444 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:54:58.494819 29449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:54:58.495550 29449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [term 1 LEADER]: Becoming Leader. State: Replica: 1f3780e55163464e82400e43a4a9018e, State: Running, Role: LEADER
I20260621 02:54:58.496016 29449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3780e55163464e82400e43a4a9018e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37105 } }
I20260621 02:54:58.496714 29444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:54:58.498867 29451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3780e55163464e82400e43a4a9018e. Latest consensus state: current_term: 1 leader_uuid: "1f3780e55163464e82400e43a4a9018e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3780e55163464e82400e43a4a9018e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37105 } } }
I20260621 02:54:58.499316 29451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:58.498867 29450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f3780e55163464e82400e43a4a9018e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3780e55163464e82400e43a4a9018e" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37105 } } }
I20260621 02:54:58.501220 29450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e [sys.catalog]: This master's current role is: LEADER
I20260621 02:54:58.503311 29455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:54:58.508514 29455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:54:58.517164 29455 catalog_manager.cc:1347] Generated new cluster ID: dd62896ee5a94f44a891769a08fdaf94
I20260621 02:54:58.517426 29455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:54:58.527575 29455 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:54:58.529328 29455 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:54:58.538954 29455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e: Generated new TSK 0
I20260621 02:54:58.540021 29455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:54:58.552152 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37105
--builtin_ntp_servers=127.11.109.20:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:54:58.887140 29468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:58.887393 29468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:58.887571 29468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:58.894892 29468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:58.894984 29468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:58.895167 29468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:54:58.904909 29468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:58.906219 29468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:58.907956 29468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:58.920928 29473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:58.921031 29476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:58.921161 29474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:58.922183 29468 server_base.cc:1034] running on GCE node
I20260621 02:54:58.922782 29468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:58.923695 29468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:58.924932 29468 hybrid_clock.cc:648] HybridClock initialized: now 1782010498924892 us; error 66 us; skew 500 ppm
I20260621 02:54:58.925314 29468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:54:58.927685 29468 webserver.cc:460] Webserver started at http://127.11.109.1:43329/ using document root <none> and password file <none>
I20260621 02:54:58.928364 29468 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:58.928498 29468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:58.928807 29468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:58.931550 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "13d3e4d2418d4553b4d6091595659d67"
format_stamp: "Formatted at 2026-06-21 02:54:58 on dist-test-slave-bc52"
server_key: "90ad99244ac77a9dcf638b67d7512a02"
server_key_iv: "a7944f7c967be6538b4c5683d147c5fa"
server_key_version: "encryptionkey@0"
I20260621 02:54:58.932377 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "13d3e4d2418d4553b4d6091595659d67"
format_stamp: "Formatted at 2026-06-21 02:54:58 on dist-test-slave-bc52"
server_key: "90ad99244ac77a9dcf638b67d7512a02"
server_key_iv: "a7944f7c967be6538b4c5683d147c5fa"
server_key_version: "encryptionkey@0"
I20260621 02:54:58.937129 29468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260621 02:54:58.940371 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:58.941661 29468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:54:58.941847 29468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "13d3e4d2418d4553b4d6091595659d67"
format_stamp: "Formatted at 2026-06-21 02:54:58 on dist-test-slave-bc52"
server_key: "90ad99244ac77a9dcf638b67d7512a02"
server_key_iv: "a7944f7c967be6538b4c5683d147c5fa"
server_key_version: "encryptionkey@0"
I20260621 02:54:58.942054 29468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:58.954005 29468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:58.954730 29468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:58.955020 29468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:58.955914 29468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:58.957327 29468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:58.957448 29468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:58.957589 29468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:58.957683 29468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:58.993988 29468 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:35711
I20260621 02:54:58.994043 29594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:35711 every 8 connection(s)
I20260621 02:54:58.995455 29468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:54:58.999198 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29468
I20260621 02:54:58.999426 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:54:59.000293 11700 external_mini_cluster.cc:1428] Setting key ba87b30e60ed50b7e549a14dfd7b0028
I20260621 02:54:59.004065 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37105
--builtin_ntp_servers=127.11.109.20:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:54:59.015722 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:58.998827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37105 (local address 127.11.109.1:53789)
0621 02:54:59.001083 (+ 2256us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:59.001373 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:59.004390 (+ 3017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:59.005227 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:59.005242 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:59.005372 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:59.006436 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.006461 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.008440 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.008449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.010291 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.010303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.011700 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.011705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.011901 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.012814 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:54:59.012817 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:59.012856 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:54:59.012932 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:59.013362 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:59.013388 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:59.013397 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:54:59.013622 (+ 225us) client_negotiation.cc:240] Negotiation successful
0621 02:54:59.014201 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"thread_start_us":1509,"threads_started":1}
I20260621 02:54:59.016810 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:58.998269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:53789 (local address 127.11.109.62:37105)
0621 02:54:58.998733 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:58.998737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.001966 (+ 3229us) server_negotiation.cc:407] Connection header received
0621 02:54:59.004726 (+ 2760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.004735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.004824 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.004943 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:59.006656 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.008026 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.010446 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.011273 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.011435 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.013136 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:59.013144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:59.013148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:54:59.013199 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:59.013254 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:54:59.013273 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:59.013363 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.016466 (+ 3103us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.016569 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":244,"threads_started":1}
I20260621 02:54:59.018188 29595 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37105
I20260621 02:54:59.018510 29595 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:59.019393 29595 heartbeater.cc:511] Master 127.11.109.62:37105 requested a full tablet report, sending...
I20260621 02:54:59.021832 29409 ts_manager.cc:194] Registered new tserver with Master: 13d3e4d2418d4553b4d6091595659d67 (127.11.109.1:35711)
I20260621 02:54:59.024025 29409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.1:53789
W20260621 02:54:59.339668 29599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:59.339958 29599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:59.340142 29599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:59.347877 29599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:59.348014 29599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:59.348199 29599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:54:59.356786 29599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:59.357975 29599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:59.359647 29599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:59.372491 29607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:59.372462 29604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:59.372519 29605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:59.373641 29599 server_base.cc:1034] running on GCE node
I20260621 02:54:59.374209 29599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:59.375036 29599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:59.376245 29599 hybrid_clock.cc:648] HybridClock initialized: now 1782010499376197 us; error 44 us; skew 500 ppm
I20260621 02:54:59.376497 29599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:54:59.378523 29599 webserver.cc:460] Webserver started at http://127.11.109.2:43387/ using document root <none> and password file <none>
I20260621 02:54:59.379099 29599 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:59.379179 29599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:59.379411 29599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:59.382066 29599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "6570334f25614280b303d6ad5646bcae"
format_stamp: "Formatted at 2026-06-21 02:54:59 on dist-test-slave-bc52"
server_key: "d5bd3d286a78334fb41ede863a617790"
server_key_iv: "61c7662e25b6ba6d4a34cb93288bb72e"
server_key_version: "encryptionkey@0"
I20260621 02:54:59.382755 29599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "6570334f25614280b303d6ad5646bcae"
format_stamp: "Formatted at 2026-06-21 02:54:59 on dist-test-slave-bc52"
server_key: "d5bd3d286a78334fb41ede863a617790"
server_key_iv: "61c7662e25b6ba6d4a34cb93288bb72e"
server_key_version: "encryptionkey@0"
I20260621 02:54:59.387392 29599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:54:59.390503 29613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:59.391667 29599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:54:59.391801 29599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "6570334f25614280b303d6ad5646bcae"
format_stamp: "Formatted at 2026-06-21 02:54:59 on dist-test-slave-bc52"
server_key: "d5bd3d286a78334fb41ede863a617790"
server_key_iv: "61c7662e25b6ba6d4a34cb93288bb72e"
server_key_version: "encryptionkey@0"
I20260621 02:54:59.391976 29599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:59.407827 29599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:59.408495 29599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:59.408733 29599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:59.409665 29599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:59.410931 29599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:59.411016 29599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:59.411091 29599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:59.411144 29599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:59.443835 29599 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:37335
I20260621 02:54:59.443887 29725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:37335 every 8 connection(s)
I20260621 02:54:59.445328 29599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:54:59.452643 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29599
I20260621 02:54:59.452829 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:54:59.453389 11700 external_mini_cluster.cc:1428] Setting key ff971702405219659e34f4ac104b5dba
I20260621 02:54:59.456429 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37105
--builtin_ntp_servers=127.11.109.20:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:54:59.458591 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.447891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:38977 (local address 127.11.109.62:37105)
0621 02:54:59.448068 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:59.448073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.449463 (+ 1390us) server_negotiation.cc:407] Connection header received
0621 02:54:59.451262 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.451267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.451331 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.451416 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:59.452439 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.453563 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.455725 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.456113 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.456253 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.457680 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:59.457692 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:59.457695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:54:59.457712 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:59.457770 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:54:59.457786 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:59.457875 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.458204 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.458306 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:54:59.459838 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.448358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37105 (local address 127.11.109.2:38977)
0621 02:54:59.449306 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:59.449342 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:59.451013 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:59.451611 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:59.451618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:59.451721 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:59.452254 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.452270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.453731 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.453734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.455574 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.455586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.456338 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.456345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.456594 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.457390 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:54:59.457393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:59.457430 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:54:59.457497 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:59.457908 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:59.457922 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:59.457935 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 02:54:59.458155 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 02:54:59.458560 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":267,"threads_started":1}
I20260621 02:54:59.461334 29726 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37105
I20260621 02:54:59.461612 29726 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:59.462388 29726 heartbeater.cc:511] Master 127.11.109.62:37105 requested a full tablet report, sending...
I20260621 02:54:59.463909 29409 ts_manager.cc:194] Registered new tserver with Master: 6570334f25614280b303d6ad5646bcae (127.11.109.2:37335)
I20260621 02:54:59.465029 29409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.2:38977
W20260621 02:54:59.780365 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:54:59.780614 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:54:59.780797 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:54:59.789955 29729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:54:59.790043 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:54:59.790241 29729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:54:59.799657 29729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:54:59.800884 29729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:54:59.802770 29729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:54:59.815254 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:59.815389 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:54:59.815316 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:54:59.816526 29729 server_base.cc:1034] running on GCE node
I20260621 02:54:59.817047 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:54:59.817955 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:54:59.819144 29729 hybrid_clock.cc:648] HybridClock initialized: now 1782010499819099 us; error 41 us; skew 500 ppm
I20260621 02:54:59.819447 29729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:54:59.821619 29729 webserver.cc:460] Webserver started at http://127.11.109.3:35407/ using document root <none> and password file <none>
I20260621 02:54:59.822273 29729 fs_manager.cc:359] Metadata directory not provided
I20260621 02:54:59.822381 29729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:54:59.822759 29729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:54:59.825515 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "667659ac30894f198e0526c59b379aa7"
format_stamp: "Formatted at 2026-06-21 02:54:59 on dist-test-slave-bc52"
server_key: "156b5ead72b81f5622497d6792953847"
server_key_iv: "f2d02e8a65131bd76bf345f6bb7d34d9"
server_key_version: "encryptionkey@0"
I20260621 02:54:59.826382 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "667659ac30894f198e0526c59b379aa7"
format_stamp: "Formatted at 2026-06-21 02:54:59 on dist-test-slave-bc52"
server_key: "156b5ead72b81f5622497d6792953847"
server_key_iv: "f2d02e8a65131bd76bf345f6bb7d34d9"
server_key_version: "encryptionkey@0"
I20260621 02:54:59.831166 29729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 02:54:59.834446 29743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:59.835641 29729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:54:59.835819 29729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "667659ac30894f198e0526c59b379aa7"
format_stamp: "Formatted at 2026-06-21 02:54:59 on dist-test-slave-bc52"
server_key: "156b5ead72b81f5622497d6792953847"
server_key_iv: "f2d02e8a65131bd76bf345f6bb7d34d9"
server_key_version: "encryptionkey@0"
I20260621 02:54:59.836020 29729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:54:59.853632 29729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:54:59.854364 29729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:54:59.854700 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:54:59.855607 29729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:54:59.856957 29729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:54:59.857090 29729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:59.857267 29729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:54:59.857365 29729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:54:59.889202 29729 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:40939
I20260621 02:54:59.889298 29855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:40939 every 8 connection(s)
I20260621 02:54:59.890522 29729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:54:59.894791 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29729
I20260621 02:54:59.894995 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:54:59.895545 11700 external_mini_cluster.cc:1428] Setting key 3f4174875892357c0863574db8bf126d
I20260621 02:54:59.903630 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.892919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:56009 (local address 127.11.109.62:37105)
0621 02:54:59.893137 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:59.893143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.894511 (+ 1368us) server_negotiation.cc:407] Connection header received
0621 02:54:59.896967 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.896973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.897084 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.897237 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:59.898597 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.899354 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.900891 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.901367 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.901497 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.902793 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:59.902801 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:59.902809 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:54:59.902825 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:59.902853 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:54:59.902864 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:59.902960 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.903173 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.903260 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:54:59.904557 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.893471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:37105 (local address 127.11.109.3:56009)
0621 02:54:59.894316 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:59.894351 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:59.896672 (+ 2321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:59.897459 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:59.897472 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:59.897600 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:54:59.898416 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.898447 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.899503 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.899508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.900748 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.900760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.901573 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.901579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.901804 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.902427 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:54:59.902429 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:54:59.902466 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:54:59.902578 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:54:59.902958 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:54:59.902970 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:54:59.902979 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:54:59.903164 (+ 185us) client_negotiation.cc:240] Negotiation successful
0621 02:54:59.903560 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":254,"threads_started":1}
I20260621 02:54:59.905871 29856 heartbeater.cc:347] Connected to a master server at 127.11.109.62:37105
I20260621 02:54:59.906169 29856 heartbeater.cc:464] Registering TS with master...
I20260621 02:54:59.906885 29856 heartbeater.cc:511] Master 127.11.109.62:37105 requested a full tablet report, sending...
I20260621 02:54:59.908229 29409 ts_manager.cc:194] Registered new tserver with Master: 667659ac30894f198e0526c59b379aa7 (127.11.109.3:40939)
I20260621 02:54:59.908977 29409 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.3:56009
I20260621 02:54:59.910930 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:54:59.920436 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.915072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.11.109.62:37105)
0621 02:54:59.915328 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:59.915337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.915724 (+ 387us) server_negotiation.cc:407] Connection header received
0621 02:54:59.915840 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.915844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.915913 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.915991 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:54:59.916397 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.917067 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.918441 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.918808 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.918900 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.919808 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:54:59.919812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:54:59.919815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:54:59.919830 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:54:59.919871 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:54:59.919882 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:54:59.919950 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.920062 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.920126 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:54:59.924081 29408 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:54:59.931078 29408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:54:59.955641 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.942407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:37335 (local address 127.0.0.1:41866)
0621 02:54:59.943425 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:59.943471 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:59.943664 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:59.944977 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:59.944988 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:59.945034 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:59.945580 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.945597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.948776 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.948785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.951953 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.951973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.954546 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.954556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.954884 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.954963 (+ 79us) client_negotiation.cc:769] Sending connection context
0621 02:54:59.955088 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 02:54:59.955302 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"threads_started":1}
I20260621 02:54:59.957450 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.942586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.11.109.2:37335)
0621 02:54:59.944275 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:59.944281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.944303 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:54:59.944412 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.944419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.944691 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.946175 (+ 1484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:59.946509 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.948573 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.952273 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.954267 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.956285 (+ 2018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.956669 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.956764 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.956998 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1546,"thread_start_us":1430,"threads_started":1}
I20260621 02:54:59.961514 29661 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9f0145c9b4a648947166fd97179d3 (DEFAULT_TABLE table=test-table [id=67c024442d404dbb9fbf4f8c4eed7e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:59.963069 29661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d9f0145c9b4a648947166fd97179d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:59.964973 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.944449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:55992)
0621 02:54:59.955940 (+ 11491us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:59.955961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:59.956078 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:59.958169 (+ 2091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:59.958174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:59.958201 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:59.958360 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.958370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.960175 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.960181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.962238 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.962252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.964140 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.964146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.964393 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.964428 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:54:59.964490 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 02:54:59.964594 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11349,"mutex_wait_us":43,"thread_start_us":10702,"threads_started":1}
I20260621 02:54:59.966167 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.944717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.11.109.1:35711)
0621 02:54:59.957558 (+ 12841us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:59.957569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.957602 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:54:59.957702 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.957711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.957974 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.958198 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:59.958477 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.959995 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.962548 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.963960 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.965366 (+ 1406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.965594 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.965665 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.965828 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":12968,"threads_started":1}
I20260621 02:54:59.967556 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.942602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:40939 (local address 127.0.0.1:39720)
0621 02:54:59.955421 (+ 12819us) negotiation.cc:107] Waiting for socket to connect
0621 02:54:59.955441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:54:59.955549 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:54:59.959933 (+ 4384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:54:59.959939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:54:59.959965 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:54:59.960473 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.960485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.962421 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.962426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.964571 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:54:59.964589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.966765 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.966772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:54:59.966943 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.966983 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:54:59.967081 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:54:59.967215 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12688,"mutex_wait_us":53,"thread_start_us":961,"threads_started":1}
I20260621 02:54:59.968375 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:54:59.943255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.11.109.3:40939)
0621 02:54:59.959107 (+ 15852us) server_negotiation.cc:206] Beginning negotiation
0621 02:54:59.959115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:54:59.959150 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:54:59.959254 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:54:59.959262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:54:59.959498 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:54:59.959674 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:54:59.960700 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.962281 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.964803 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:54:59.966235 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:54:59.966372 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:54:59.966639 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0621 02:54:59.967151 (+ 512us) server_negotiation.cc:299] Negotiation successful
0621 02:54:59.968004 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15651,"thread_start_us":19214,"threads_started":1}
I20260621 02:54:59.969950 29530 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9f0145c9b4a648947166fd97179d3 (DEFAULT_TABLE table=test-table [id=67c024442d404dbb9fbf4f8c4eed7e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:59.971101 29530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d9f0145c9b4a648947166fd97179d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:59.972086 29791 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9f0145c9b4a648947166fd97179d3 (DEFAULT_TABLE table=test-table [id=67c024442d404dbb9fbf4f8c4eed7e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:54:59.973551 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d9f0145c9b4a648947166fd97179d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:54:59.977159 29874 tablet_bootstrap.cc:492] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Bootstrap starting.
I20260621 02:54:59.980746 29874 tablet_bootstrap.cc:654] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:59.982029 29874 log.cc:824] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:59.984146 29876 tablet_bootstrap.cc:492] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: Bootstrap starting.
I20260621 02:54:59.985247 29874 tablet_bootstrap.cc:492] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: No bootstrap required, opened a new log
I20260621 02:54:59.985534 29874 ts_tablet_manager.cc:1397] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 02:54:59.986855 29877 tablet_bootstrap.cc:492] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Bootstrap starting.
I20260621 02:54:59.988693 29876 tablet_bootstrap.cc:654] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:59.989799 29876 log.cc:824] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:59.990288 29877 tablet_bootstrap.cc:654] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Neither blocks nor log segments found. Creating new log.
I20260621 02:54:59.991446 29877 log.cc:824] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:54:59.992779 29876 tablet_bootstrap.cc:492] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: No bootstrap required, opened a new log
I20260621 02:54:59.993098 29876 ts_tablet_manager.cc:1397] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:54:59.994477 29877 tablet_bootstrap.cc:492] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: No bootstrap required, opened a new log
I20260621 02:54:59.994710 29877 ts_tablet_manager.cc:1397] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:54:59.995867 29874 raft_consensus.cc:348] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:54:59.996322 29874 raft_consensus.cc:374] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:54:59.996436 29874 raft_consensus.cc:729] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6570334f25614280b303d6ad5646bcae, State: Initialized, Role: FOLLOWER
I20260621 02:54:59.997056 29874 consensus_queue.cc:260] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.008229 29876 raft_consensus.cc:348] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.008689 29876 raft_consensus.cc:374] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:00.008802 29876 raft_consensus.cc:729] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d3e4d2418d4553b4d6091595659d67, State: Initialized, Role: FOLLOWER
I20260621 02:55:00.008821 29877 raft_consensus.cc:348] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.009544 29876 consensus_queue.cc:260] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.010491 29877 raft_consensus.cc:374] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:00.010635 29877 raft_consensus.cc:729] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667659ac30894f198e0526c59b379aa7, State: Initialized, Role: FOLLOWER
I20260621 02:55:00.011433 29877 consensus_queue.cc:260] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.011698 29726 heartbeater.cc:503] Master 127.11.109.62:37105 was elected leader, sending a full tablet report...
I20260621 02:55:00.012570 29874 ts_tablet_manager.cc:1428] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Time spent starting tablet: real 0.027s user 0.015s sys 0.012s
I20260621 02:55:00.013603 29856 heartbeater.cc:503] Master 127.11.109.62:37105 was elected leader, sending a full tablet report...
I20260621 02:55:00.014206 29877 ts_tablet_manager.cc:1428] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260621 02:55:00.015545 29595 heartbeater.cc:503] Master 127.11.109.62:37105 was elected leader, sending a full tablet report...
I20260621 02:55:00.016192 29876 ts_tablet_manager.cc:1428] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: Time spent starting tablet: real 0.023s user 0.012s sys 0.007s
I20260621 02:55:00.073139 29881 raft_consensus.cc:482] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:55:00.073541 29881 raft_consensus.cc:504] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.075266 29881 leader_election.cc:288] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6570334f25614280b303d6ad5646bcae (127.11.109.2:37335), 667659ac30894f198e0526c59b379aa7 (127.11.109.3:40939)
I20260621 02:55:00.083802 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.075926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:38401 (local address 127.11.109.3:40939)
0621 02:55:00.076140 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:00.076146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:00.077313 (+ 1167us) server_negotiation.cc:407] Connection header received
0621 02:55:00.077470 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:00.077474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:00.077530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:00.077598 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:00.078157 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.079114 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.081507 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.082739 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.082890 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.082950 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:00.083433 (+ 483us) server_negotiation.cc:299] Negotiation successful
0621 02:55:00.083559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:55:00.083827 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.075917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:33395 (local address 127.11.109.2:37335)
0621 02:55:00.076147 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:00.076153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:00.076720 (+ 567us) server_negotiation.cc:407] Connection header received
0621 02:55:00.076986 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:00.076994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:00.077084 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:00.077213 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:00.077816 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.078789 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.081415 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.082695 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.082868 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.082940 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:00.083433 (+ 493us) server_negotiation.cc:299] Negotiation successful
0621 02:55:00.083559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:55:00.083978 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.075802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:37335 (local address 127.11.109.1:33395)
0621 02:55:00.076591 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.076607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:00.076793 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:00.077248 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:00.077254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:00.077284 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:00.077667 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:00.077677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.078922 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.078928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:00.081195 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:00.081210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.082943 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.082948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:00.083189 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.083291 (+ 102us) client_negotiation.cc:769] Sending connection context
0621 02:55:00.083385 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:55:00.083552 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":98,"thread_start_us":481,"threads_started":1}
I20260621 02:55:00.084012 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.075802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:40939 (local address 127.11.109.1:38401)
0621 02:55:00.077167 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.077187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:00.077331 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:00.077654 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:00.077658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:00.077681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:00.078001 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:00.078009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.079236 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.079242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:00.081324 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:00.081337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.082924 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.082932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:00.083195 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.083296 (+ 101us) client_negotiation.cc:769] Sending connection context
0621 02:55:00.083385 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 02:55:00.083552 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":175,"spinlock_wait_cycles":1280,"thread_start_us":1022,"threads_started":1}
I20260621 02:55:00.085090 29681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9f0145c9b4a648947166fd97179d3" candidate_uuid: "13d3e4d2418d4553b4d6091595659d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6570334f25614280b303d6ad5646bcae" is_pre_election: true
I20260621 02:55:00.085088 29811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9f0145c9b4a648947166fd97179d3" candidate_uuid: "13d3e4d2418d4553b4d6091595659d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667659ac30894f198e0526c59b379aa7" is_pre_election: true
I20260621 02:55:00.085640 29681 raft_consensus.cc:2454] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d3e4d2418d4553b4d6091595659d67 in term 0.
I20260621 02:55:00.085642 29811 raft_consensus.cc:2454] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d3e4d2418d4553b4d6091595659d67 in term 0.
I20260621 02:55:00.086308 29485 leader_election.cc:302] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13d3e4d2418d4553b4d6091595659d67, 6570334f25614280b303d6ad5646bcae, 667659ac30894f198e0526c59b379aa7; no voters:
I20260621 02:55:00.086701 29881 raft_consensus.cc:2781] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:55:00.086881 29881 raft_consensus.cc:482] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:55:00.087001 29881 raft_consensus.cc:3037] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:00.089160 29881 raft_consensus.cc:504] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.089907 29881 leader_election.cc:288] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [CANDIDATE]: Term 1 election: Requested vote from peers 6570334f25614280b303d6ad5646bcae (127.11.109.2:37335), 667659ac30894f198e0526c59b379aa7 (127.11.109.3:40939)
I20260621 02:55:00.090255 29681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9f0145c9b4a648947166fd97179d3" candidate_uuid: "13d3e4d2418d4553b4d6091595659d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6570334f25614280b303d6ad5646bcae"
I20260621 02:55:00.090402 29811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9f0145c9b4a648947166fd97179d3" candidate_uuid: "13d3e4d2418d4553b4d6091595659d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667659ac30894f198e0526c59b379aa7"
I20260621 02:55:00.090462 29681 raft_consensus.cc:3037] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:00.090605 29811 raft_consensus.cc:3037] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:00.093765 29811 raft_consensus.cc:2454] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d3e4d2418d4553b4d6091595659d67 in term 1.
I20260621 02:55:00.093782 29681 raft_consensus.cc:2454] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d3e4d2418d4553b4d6091595659d67 in term 1.
I20260621 02:55:00.094194 29485 leader_election.cc:302] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13d3e4d2418d4553b4d6091595659d67, 6570334f25614280b303d6ad5646bcae, 667659ac30894f198e0526c59b379aa7; no voters:
I20260621 02:55:00.094527 29881 raft_consensus.cc:2781] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:00.094977 29881 raft_consensus.cc:686] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 1 LEADER]: Becoming Leader. State: Replica: 13d3e4d2418d4553b4d6091595659d67, State: Running, Role: LEADER
I20260621 02:55:00.095513 29881 consensus_queue.cc:237] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } }
I20260621 02:55:00.100044 29409 catalog_manager.cc:5495] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d3e4d2418d4553b4d6091595659d67 (127.11.109.1). New cstate: current_term: 1 leader_uuid: "13d3e4d2418d4553b4d6091595659d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d3e4d2418d4553b4d6091595659d67" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 35711 } health_report { overall_health: HEALTHY } } }
I20260621 02:55:00.119827 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.114617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.11.109.1:35711)
0621 02:55:00.114856 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:00.114861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:00.114876 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:55:00.115061 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:00.115065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:00.115151 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:00.115272 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:00.115572 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.116246 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.117849 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.118238 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.118328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.118800 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:00.119259 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:00.119375 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:00.119478 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 02:55:00.119533 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260621 02:55:00.127462 29811 raft_consensus.cc:1261] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 1 FOLLOWER]: Refusing update from remote peer 13d3e4d2418d4553b4d6091595659d67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:00.127485 29681 raft_consensus.cc:1261] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 1 FOLLOWER]: Refusing update from remote peer 13d3e4d2418d4553b4d6091595659d67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:00.128342 29881 consensus_queue.cc:1035] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "667659ac30894f198e0526c59b379aa7" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:00.128656 29885 consensus_queue.cc:1035] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6570334f25614280b303d6ad5646bcae" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:00.139658 29888 mvcc.cc:205] Tried to move back new op lower bound from 7299115008511889408 to 7299115008395493376. Current Snapshot: MvccSnapshot[applied={T|T < 7299115008511889408}]
I20260621 02:55:00.152702 29407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:00.152972 29407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:00.156337 29407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f3780e55163464e82400e43a4a9018e: Sending DeleteTablet for 3 replicas of tablet f7d9f0145c9b4a648947166fd97179d3
I20260621 02:55:00.157434 29661 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9f0145c9b4a648947166fd97179d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:00 UTC) from {username='slave'} at 127.0.0.1:41866
I20260621 02:55:00.157548 29791 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9f0145c9b4a648947166fd97179d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:00 UTC) from {username='slave'} at 127.0.0.1:39720
I20260621 02:55:00.158564 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29468
I20260621 02:55:00.158452 29530 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9f0145c9b4a648947166fd97179d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:00 UTC) from {username='slave'} at 127.0.0.1:55992
I20260621 02:55:00.166703 29900 tablet_replica.cc:330] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: stopping tablet replica
I20260621 02:55:00.167181 29900 raft_consensus.cc:2229] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:55:00.167879 29900 raft_consensus.cc:2258] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:00.168650 29899 tablet_replica.cc:330] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: stopping tablet replica
I20260621 02:55:00.169025 29899 raft_consensus.cc:2229] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:00.169620 29899 raft_consensus.cc:2258] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:00.171250 29900 ts_tablet_manager.cc:1905] T f7d9f0145c9b4a648947166fd97179d3 P 13d3e4d2418d4553b4d6091595659d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:00.174407 29898 tablet_replica.cc:330] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: stopping tablet replica
I20260621 02:55:00.174914 29898 raft_consensus.cc:2229] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:00.175436 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.159767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.11.109.1:35711)
0621 02:55:00.159940 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:00.159948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:00.165976 (+ 6028us) server_negotiation.cc:407] Connection header received
0621 02:55:00.166191 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:00.166197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:00.166285 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:00.166410 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:00.168343 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.169409 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.170958 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.171866 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.172020 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.173664 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:00.173674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:00.173688 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:00.174533 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:00.174592 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:00.174627 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:00.174707 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:00.174959 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 02:55:00.175061 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:55:00.175411 29898 raft_consensus.cc:2258] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:00.176515 29590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:00.176877 29899 ts_tablet_manager.cc:1905] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:00.177853 29898 ts_tablet_manager.cc:1905] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:00.183809 29898 ts_tablet_manager.cc:1918] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:00.183981 29898 log.cc:1196] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/f7d9f0145c9b4a648947166fd97179d3
I20260621 02:55:00.184439 29898 ts_tablet_manager.cc:1939] T f7d9f0145c9b4a648947166fd97179d3 P 6570334f25614280b303d6ad5646bcae: Deleting consensus metadata
I20260621 02:55:00.184755 29899 ts_tablet_manager.cc:1918] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:00.184906 29899 log.cc:1196] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/f7d9f0145c9b4a648947166fd97179d3
I20260621 02:55:00.185456 29899 ts_tablet_manager.cc:1939] T f7d9f0145c9b4a648947166fd97179d3 P 667659ac30894f198e0526c59b379aa7: Deleting consensus metadata
I20260621 02:55:00.186152 29394 catalog_manager.cc:4841] TS 6570334f25614280b303d6ad5646bcae (127.11.109.2:37335): tablet f7d9f0145c9b4a648947166fd97179d3 (table test-table [id=67c024442d404dbb9fbf4f8c4eed7e4d]) successfully deleted
I20260621 02:55:00.187116 29395 catalog_manager.cc:4841] TS 667659ac30894f198e0526c59b379aa7 (127.11.109.3:40939): tablet f7d9f0145c9b4a648947166fd97179d3 (table test-table [id=67c024442d404dbb9fbf4f8c4eed7e4d]) successfully deleted
I20260621 02:55:00.276098 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29468
W20260621 02:55:00.295267 29395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.109.1:35711 (error 108)
I20260621 02:55:00.295346 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29599
I20260621 02:55:00.297449 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.296855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56032)
0621 02:55:00.297103 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.297250 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 02:55:00.297701 29395 catalog_manager.cc:4584] TS 13d3e4d2418d4553b4d6091595659d67 (127.11.109.1:35711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7d9f0145c9b4a648947166fd97179d3: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
I20260621 02:55:00.302546 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.296218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.11.109.2:37335)
0621 02:55:00.296461 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:00.296467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:00.296626 (+ 159us) server_negotiation.cc:407] Connection header received
0621 02:55:00.296849 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:00.296855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:00.296982 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:00.297111 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:00.297598 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.298459 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.299718 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.300097 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.300200 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.301131 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:00.301140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:00.301153 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:00.301875 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:00.301920 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:00.301939 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:00.302031 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:00.302150 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 02:55:00.302248 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260621 02:55:00.303285 29721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:00.351279 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.350840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56048)
0621 02:55:00.351043 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.351123 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 02:55:00.352293 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.351959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56054)
0621 02:55:00.352113 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.352176 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:55:00.391350 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29599
I20260621 02:55:00.408141 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29729
I20260621 02:55:00.412448 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.412029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56066)
0621 02:55:00.412222 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.412296 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:55:00.413748 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.413181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56080)
0621 02:55:00.413475 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.413579 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:55:00.415468 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.408918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.11.109.3:40939)
0621 02:55:00.409102 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:00.409111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:00.409133 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:55:00.409396 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:00.409400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:00.409497 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:00.409609 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:00.410007 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.410744 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.412053 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:00.412439 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:00.412570 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:00.413910 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:00.413925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:00.413950 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:00.414686 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:00.414757 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:00.414798 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:00.414933 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:00.415108 (+ 175us) server_negotiation.cc:299] Negotiation successful
0621 02:55:00.415228 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:55:00.416096 29851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:00.494469 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.493931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56090)
0621 02:55:00.494160 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.494272 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 02:55:00.495551 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:00.495162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:35711 (local address 127.0.0.1:56094)
0621 02:55:00.495376 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:00.495445 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:35711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260621 02:55:00.504031 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29729
I20260621 02:55:00.520736 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29378
I20260621 02:55:00.521823 29439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:00.629798 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29378
2026-06-21T02:55:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2651 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:00 dist-test-slave-bc52 krb5kdc[29909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:00 dist-test-slave-bc52 krb5kdc[29909](info): set up 2 sockets
Jun 21 02:55:00 dist-test-slave-bc52 krb5kdc[29909](info): commencing operation
krb5kdc: starting...
W20260621 02:55:01.729395 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:01 dist-test-slave-bc52 krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:01.960340 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:33401
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:02.277634 29922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:02.277973 29922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:02.278151 29922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:02.285297 29922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:02.285426 29922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:02.285542 29922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:02.285660 29922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:02.285760 29922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:02.296713 29922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:02.298066 29922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:02.299310 29922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:02.310698 29927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:02.310660 29930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:02.310750 29928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:02.311772 29922 server_base.cc:1034] running on GCE node
I20260621 02:55:02.312536 29922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:02.313879 29922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:02.315106 29922 hybrid_clock.cc:648] HybridClock initialized: now 1782010502315076 us; error 42 us; skew 500 ppm
I20260621 02:55:02.315443 29922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:02 dist-test-slave-bc52 krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:02.318468 29922 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:02.319885 29922 webserver.cc:460] Webserver started at http://127.11.109.62:35269/ using document root <none> and password file <none>
I20260621 02:55:02.320530 29922 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:02.320675 29922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:02.320966 29922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:02.323792 29922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "ff50b2573248442a828647124d42a2e1"
format_stamp: "Formatted at 2026-06-21 02:55:02 on dist-test-slave-bc52"
server_key: "6dd16e63e2e6248f436820adceeb565c"
server_key_iv: "55a95ed8064ec48ec4212f04edcd43e9"
server_key_version: "encryptionkey@0"
I20260621 02:55:02.324541 29922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "ff50b2573248442a828647124d42a2e1"
format_stamp: "Formatted at 2026-06-21 02:55:02 on dist-test-slave-bc52"
server_key: "6dd16e63e2e6248f436820adceeb565c"
server_key_iv: "55a95ed8064ec48ec4212f04edcd43e9"
server_key_version: "encryptionkey@0"
I20260621 02:55:02.329583 29922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260621 02:55:02.332712 29937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:02.334035 29922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:02.334218 29922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "ff50b2573248442a828647124d42a2e1"
format_stamp: "Formatted at 2026-06-21 02:55:02 on dist-test-slave-bc52"
server_key: "6dd16e63e2e6248f436820adceeb565c"
server_key_iv: "55a95ed8064ec48ec4212f04edcd43e9"
server_key_version: "encryptionkey@0"
I20260621 02:55:02.334411 29922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:02.361438 29922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:02.365299 29922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:02.365583 29922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:02.383478 29922 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:33401
I20260621 02:55:02.383496 29988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:33401 every 8 connection(s)
I20260621 02:55:02.384877 29922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:02.388423 29989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:02.391999 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 29922
I20260621 02:55:02.392167 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:02.392726 11700 external_mini_cluster.cc:1428] Setting key 47fb4449c8cc0ea569420a87e4c17c76
I20260621 02:55:02.400962 29989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1: Bootstrap starting.
I20260621 02:55:02.404580 29989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:02.405910 29989 log.cc:824] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:55:02 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:02.408705 29989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1: No bootstrap required, opened a new log
I20260621 02:55:02.416618 29992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:02.394595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.11.109.62:33401)
0621 02:55:02.395353 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:02.395367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:02.395406 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:55:02.396962 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:02.396981 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:02.397420 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:02.397725 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:02.399489 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:02.400702 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:02.402566 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:02.403230 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:02.403350 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:02.408185 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:02.408196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:02.408212 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:02.408269 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:02.412311 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:02.412980 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:02.412988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:02.412996 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:02.413238 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:02.413588 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:02.413593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:02.413596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:02.414217 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:02.414543 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:02.414867 (+ 324us) server_negotiation.cc:299] Negotiation successful
0621 02:55:02.415330 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":360,"threads_started":1}
I20260621 02:55:02.419656 29989 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff50b2573248442a828647124d42a2e1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33401 } }
I20260621 02:55:02.419998 29989 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:02.420069 29989 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff50b2573248442a828647124d42a2e1, State: Initialized, Role: FOLLOWER
I20260621 02:55:02.420595 29989 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff50b2573248442a828647124d42a2e1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33401 } }
I20260621 02:55:02.420802 29989 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:02.420897 29989 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:02.421015 29989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:02.423132 29989 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff50b2573248442a828647124d42a2e1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33401 } }
I20260621 02:55:02.423738 29989 leader_election.cc:302] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff50b2573248442a828647124d42a2e1; no voters:
I20260621 02:55:02.424325 29989 leader_election.cc:288] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:02.424469 29994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:02.425117 29994 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [term 1 LEADER]: Becoming Leader. State: Replica: ff50b2573248442a828647124d42a2e1, State: Running, Role: LEADER
I20260621 02:55:02.425659 29994 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff50b2573248442a828647124d42a2e1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33401 } }
I20260621 02:55:02.426298 29989 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:02.428730 29996 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff50b2573248442a828647124d42a2e1. Latest consensus state: current_term: 1 leader_uuid: "ff50b2573248442a828647124d42a2e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff50b2573248442a828647124d42a2e1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33401 } } }
I20260621 02:55:02.429101 29996 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:02.428819 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff50b2573248442a828647124d42a2e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff50b2573248442a828647124d42a2e1" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33401 } } }
I20260621 02:55:02.429240 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:02.434144 30001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:02.438244 30001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:02.446480 30001 catalog_manager.cc:1347] Generated new cluster ID: ec290d7600734053815b2fe83069a5ff
I20260621 02:55:02.446599 30001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:02.463676 30001 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:02.464977 30001 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:02.477753 30001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1: Generated new TSK 0
I20260621 02:55:02.478530 30001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:55:02.546063 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:33401
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:02.864909 30015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:02.865226 30015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:02.865432 30015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:02.872750 30015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:02.872867 30015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:02.872973 30015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:55:02.882812 30015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:02.884070 30015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:02.885409 30015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:02.898452 30020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:02.898521 30023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:02.898418 30021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:02.899358 30015 server_base.cc:1034] running on GCE node
I20260621 02:55:02.899966 30015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:02.900897 30015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:02.902124 30015 hybrid_clock.cc:648] HybridClock initialized: now 1782010502902089 us; error 48 us; skew 500 ppm
I20260621 02:55:02.902539 30015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:02 dist-test-slave-bc52 krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:02.905594 30015 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:55:02.907124 30015 webserver.cc:460] Webserver started at http://127.11.109.1:46681/ using document root <none> and password file <none>
I20260621 02:55:02.907821 30015 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:02.907917 30015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:02.908157 30015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:02.911024 30015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "88beb88434ff4bc1991d3b21f522bda4"
format_stamp: "Formatted at 2026-06-21 02:55:02 on dist-test-slave-bc52"
server_key: "e622bd61aa0d711955e9fe4d1426a5b2"
server_key_iv: "dd4b57aa97dbc7ca78db3f539e898e8d"
server_key_version: "encryptionkey@0"
I20260621 02:55:02.911688 30015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "88beb88434ff4bc1991d3b21f522bda4"
format_stamp: "Formatted at 2026-06-21 02:55:02 on dist-test-slave-bc52"
server_key: "e622bd61aa0d711955e9fe4d1426a5b2"
server_key_iv: "dd4b57aa97dbc7ca78db3f539e898e8d"
server_key_version: "encryptionkey@0"
I20260621 02:55:02.916584 30015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:55:02.919958 30030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:02.921337 30015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:02.921490 30015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "88beb88434ff4bc1991d3b21f522bda4"
format_stamp: "Formatted at 2026-06-21 02:55:02 on dist-test-slave-bc52"
server_key: "e622bd61aa0d711955e9fe4d1426a5b2"
server_key_iv: "dd4b57aa97dbc7ca78db3f539e898e8d"
server_key_version: "encryptionkey@0"
I20260621 02:55:02.921675 30015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:02.947939 30015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:02.951836 30015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:02.952122 30015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:02.953039 30015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:02.954331 30015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:02.954419 30015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:02.954593 30015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:02.954653 30015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:02.986851 30015 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:36403
I20260621 02:55:02.986930 30142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:36403 every 8 connection(s)
I20260621 02:55:02.988142 30015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:55:02.991963 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30015
I20260621 02:55:02.992136 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:55:02.992681 11700 external_mini_cluster.cc:1428] Setting key cc08974b80275b337fc3d4673e0c8f98
Jun 21 02:55:02 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010502, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:03.009333 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:02.990575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:59485 (local address 127.11.109.62:33401)
0621 02:55:02.991009 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:02.991014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:02.991875 (+ 861us) server_negotiation.cc:407] Connection header received
0621 02:55:02.994046 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:02.994051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:02.994121 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:02.994230 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:02.996109 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:02.996798 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:02.999210 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:02.999604 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:02.999695 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:03.003825 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:03.003829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:03.003832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:03.003864 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:03.006340 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:03.007047 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:03.007058 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:03.007061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:03.007173 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:03.007610 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:03.007615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:03.007618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:03.007881 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:03.008037 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:03.008703 (+ 666us) server_negotiation.cc:299] Negotiation successful
0621 02:55:03.008936 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":240,"threads_started":1}
I20260621 02:55:03.010241 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:02.990973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:33401 (local address 127.11.109.1:59485)
0621 02:55:02.991676 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:02.991737 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:02.993793 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:02.994432 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:02.994441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:02.995125 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:02.995928 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:02.995956 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:02.996958 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:02.996966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:02.999067 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:02.999091 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:02.999858 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:02.999867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:03.000190 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:03.000953 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:03.000985 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:03.003611 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:03.006476 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:03.006484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:03.006500 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:03.006919 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:03.007310 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:03.007315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:03.007318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:03.007491 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:03.008008 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:03.008015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:03.008336 (+ 321us) client_negotiation.cc:769] Sending connection context
0621 02:55:03.008587 (+ 251us) client_negotiation.cc:240] Negotiation successful
0621 02:55:03.009082 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260621 02:55:03.011890 30143 heartbeater.cc:347] Connected to a master server at 127.11.109.62:33401
I20260621 02:55:03.012183 30143 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:03.012887 30143 heartbeater.cc:511] Master 127.11.109.62:33401 requested a full tablet report, sending...
I20260621 02:55:03.014997 29954 ts_manager.cc:194] Registered new tserver with Master: 88beb88434ff4bc1991d3b21f522bda4 (127.11.109.1:36403)
I20260621 02:55:03.016389 29954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:59485
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:55:03.047683 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:33401
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:03.364290 30149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:03.364527 30149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:03.364616 30149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:03.371392 30149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:03.371477 30149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:03.371577 30149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:55:03.380461 30149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:03.381501 30149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:03.382680 30149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:03.394893 30154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:03.394862 30155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:03.395681 30157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:03.395953 30149 server_base.cc:1034] running on GCE node
I20260621 02:55:03.396579 30149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:03.397562 30149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:03.398775 30149 hybrid_clock.cc:648] HybridClock initialized: now 1782010503398721 us; error 41 us; skew 500 ppm
I20260621 02:55:03.399019 30149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:03 dist-test-slave-bc52 krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010503, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:03.401993 30149 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:55:03.403426 30149 webserver.cc:460] Webserver started at http://127.11.109.2:44659/ using document root <none> and password file <none>
I20260621 02:55:03.404057 30149 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:03.404182 30149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:03.404487 30149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:03.407280 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "dde213152bf44f37bbe17d3330bda849"
format_stamp: "Formatted at 2026-06-21 02:55:03 on dist-test-slave-bc52"
server_key: "7f5d7fc5aa603f9ee92c344d101309f3"
server_key_iv: "3e399b5f056805b94467c15142fc0f8b"
server_key_version: "encryptionkey@0"
I20260621 02:55:03.408010 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "dde213152bf44f37bbe17d3330bda849"
format_stamp: "Formatted at 2026-06-21 02:55:03 on dist-test-slave-bc52"
server_key: "7f5d7fc5aa603f9ee92c344d101309f3"
server_key_iv: "3e399b5f056805b94467c15142fc0f8b"
server_key_version: "encryptionkey@0"
I20260621 02:55:03.412910 30149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260621 02:55:03.416329 30164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:03.417809 30149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:03.417973 30149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "dde213152bf44f37bbe17d3330bda849"
format_stamp: "Formatted at 2026-06-21 02:55:03 on dist-test-slave-bc52"
server_key: "7f5d7fc5aa603f9ee92c344d101309f3"
server_key_iv: "3e399b5f056805b94467c15142fc0f8b"
server_key_version: "encryptionkey@0"
I20260621 02:55:03.418166 30149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:03.449411 30149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:03.453589 30149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:03.453917 30149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:03.454870 30149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:03.456161 30149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:03.456290 30149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:03.456445 30149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:03.456529 30149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:03.488768 30149 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:36225
I20260621 02:55:03.488814 30276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:36225 every 8 connection(s)
I20260621 02:55:03.490069 30149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:55:03.498730 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30149
I20260621 02:55:03.498911 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:55:03.499347 11700 external_mini_cluster.cc:1428] Setting key 557755ef804a15b4c3061e673a3923d9
Jun 21 02:55:03 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010503, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:03.510890 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:03.492423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:40217 (local address 127.11.109.62:33401)
0621 02:55:03.492591 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:03.492595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:03.493645 (+ 1050us) server_negotiation.cc:407] Connection header received
0621 02:55:03.495369 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:03.495374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:03.495467 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:03.495591 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:03.497096 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:03.497869 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:03.499560 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:03.500138 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:03.500288 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:03.504184 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:03.504200 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:03.504208 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:03.504252 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:03.507816 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:03.508495 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:03.508502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:03.508508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:03.508664 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:03.509066 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:03.509074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:03.509079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:03.509465 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:03.509654 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:03.510248 (+ 594us) server_negotiation.cc:299] Negotiation successful
0621 02:55:03.510501 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:55:03.511917 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:03.492794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:33401 (local address 127.11.109.2:40217)
0621 02:55:03.493490 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:03.493530 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:03.495108 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:03.495811 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:03.495819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:03.496407 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:03.496927 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:03.496943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:03.498041 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:03.498045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:03.499376 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:03.499387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:03.500370 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:03.500379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:03.500653 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:03.501424 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:03.501462 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:03.503972 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:03.507954 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:03.507960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:03.507971 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:03.508375 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:03.508773 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:03.508776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:03.508778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:03.508960 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:03.509608 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:03.509613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:03.509885 (+ 272us) client_negotiation.cc:769] Sending connection context
0621 02:55:03.510110 (+ 225us) client_negotiation.cc:240] Negotiation successful
0621 02:55:03.510469 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":325,"threads_started":1}
I20260621 02:55:03.513393 30277 heartbeater.cc:347] Connected to a master server at 127.11.109.62:33401
I20260621 02:55:03.513681 30277 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:03.514508 30277 heartbeater.cc:511] Master 127.11.109.62:33401 requested a full tablet report, sending...
I20260621 02:55:03.516106 29954 ts_manager.cc:194] Registered new tserver with Master: dde213152bf44f37bbe17d3330bda849 (127.11.109.2:36225)
I20260621 02:55:03.516907 29954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:40217
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:55:03.557197 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:33401
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:03.876655 30282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:03.876919 30282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:03.877070 30282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:03.884039 30282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:03.884142 30282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:03.884341 30282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:55:03.893523 30282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:33401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:03.894696 30282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:03.895951 30282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:03.908897 30290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:03.908978 30287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:03.908926 30288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:03.909829 30282 server_base.cc:1034] running on GCE node
I20260621 02:55:03.910458 30282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:03.911321 30282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:03.912557 30282 hybrid_clock.cc:648] HybridClock initialized: now 1782010503912521 us; error 50 us; skew 500 ppm
I20260621 02:55:03.912865 30282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:03 dist-test-slave-bc52 krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010503, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:03.916050 30282 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:55:03.917479 30282 webserver.cc:460] Webserver started at http://127.11.109.3:39595/ using document root <none> and password file <none>
I20260621 02:55:03.918147 30282 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:03.918289 30282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:03.918594 30282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:03.921545 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "33d85b6a76bb43ac96cdec9f4db9f359"
format_stamp: "Formatted at 2026-06-21 02:55:03 on dist-test-slave-bc52"
server_key: "0291bf855e0b831e2a34aea3ae7238f7"
server_key_iv: "308864d73fced4c4e11b986e68ec632b"
server_key_version: "encryptionkey@0"
I20260621 02:55:03.922387 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "33d85b6a76bb43ac96cdec9f4db9f359"
format_stamp: "Formatted at 2026-06-21 02:55:03 on dist-test-slave-bc52"
server_key: "0291bf855e0b831e2a34aea3ae7238f7"
server_key_iv: "308864d73fced4c4e11b986e68ec632b"
server_key_version: "encryptionkey@0"
I20260621 02:55:03.927520 30282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:55:03.930845 30297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:03.932161 30282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:55:03.932339 30282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "33d85b6a76bb43ac96cdec9f4db9f359"
format_stamp: "Formatted at 2026-06-21 02:55:03 on dist-test-slave-bc52"
server_key: "0291bf855e0b831e2a34aea3ae7238f7"
server_key_iv: "308864d73fced4c4e11b986e68ec632b"
server_key_version: "encryptionkey@0"
I20260621 02:55:03.932551 30282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:03.952078 30282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:03.955736 30282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:03.956022 30282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:03.956913 30282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:03.958195 30282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:03.958298 30282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:03.958439 30282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:03.958516 30282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:03.990952 30282 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:45015
I20260621 02:55:03.990958 30409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:45015 every 8 connection(s)
I20260621 02:55:03.992415 30282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:55:03.999384 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30282
I20260621 02:55:03.999573 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:55:04.000022 11700 external_mini_cluster.cc:1428] Setting key 28bb95af7421a934001e8489845812dd
Jun 21 02:55:04 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010503, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:04.011534 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:03.995093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:37401 (local address 127.11.109.62:33401)
0621 02:55:03.995327 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:03.995334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:03.996255 (+ 921us) server_negotiation.cc:407] Connection header received
0621 02:55:03.998042 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:03.998047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:03.998110 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:03.998193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:03.999819 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.000658 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.002759 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.003141 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.003229 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.006659 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:04.006664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:04.006666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:04.006695 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:04.009157 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.009808 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.009811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.009814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.009927 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.010253 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.010256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.010258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.010485 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:04.010585 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.011129 (+ 544us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.011272 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:55:04.012468 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:03.995441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:33401 (local address 127.11.109.3:37401)
0621 02:55:03.996104 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:03.996138 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:03.997799 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:03.998375 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:03.998383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:03.998865 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:03.999642 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:03.999666 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.000816 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.000820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.002622 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.002637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.003329 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.003333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.003501 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.004182 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:04.004203 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:04.006466 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:04.009338 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:04.009344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:04.009354 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:04.009708 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:04.010010 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:04.010013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:04.010015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:04.010164 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:04.010582 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:04.010587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:04.010849 (+ 262us) client_negotiation.cc:769] Sending connection context
0621 02:55:04.011040 (+ 191us) client_negotiation.cc:240] Negotiation successful
0621 02:55:04.011375 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260621 02:55:04.014014 30410 heartbeater.cc:347] Connected to a master server at 127.11.109.62:33401
I20260621 02:55:04.014326 30410 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:04.014983 30410 heartbeater.cc:511] Master 127.11.109.62:33401 requested a full tablet report, sending...
I20260621 02:55:04.016304 29954 ts_manager.cc:194] Registered new tserver with Master: 33d85b6a76bb43ac96cdec9f4db9f359 (127.11.109.3:45015)
I20260621 02:55:04.017504 29954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:37401
I20260621 02:55:04.019336 30143 heartbeater.cc:503] Master 127.11.109.62:33401 was elected leader, sending a full tablet report...
I20260621 02:55:04.027000 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:55:04.043452 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.031226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.11.109.62:33401)
0621 02:55:04.031444 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.031449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.031736 (+ 287us) server_negotiation.cc:407] Connection header received
0621 02:55:04.031917 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.031921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.031976 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.032048 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:04.033526 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.034230 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.035464 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.035866 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.035956 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.038648 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:04.038653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:04.038656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:04.038684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:04.041123 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.041839 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.041843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.041845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.041958 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.042333 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.042336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.042338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.042627 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:04.042801 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.043104 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.043222 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:55:04.047381 29954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:55:04.054056 29954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:55:04.107831 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.066709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.11.109.1:36403)
0621 02:55:04.098368 (+ 31659us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.098380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.098416 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:55:04.098532 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.098543 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.098843 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.099060 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:04.099897 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.102618 (+ 2721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.105083 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.106549 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.106687 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.106879 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.107338 (+ 459us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.107492 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31418,"thread_start_us":32561,"threads_started":1}
I20260621 02:55:04.107882 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.066117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:36403 (local address 127.0.0.1:45208)
0621 02:55:04.093286 (+ 27169us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:04.093306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:04.093439 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:04.099146 (+ 5707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:04.099156 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:04.099194 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:04.099664 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.099683 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.102785 (+ 3102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.102792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.104830 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.104852 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.106978 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.106983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.107110 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.107142 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:55:04.107286 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 02:55:04.107469 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27008,"mutex_wait_us":3,"thread_start_us":35304,"threads_started":1}
I20260621 02:55:04.109069 30425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.065916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:45015 (local address 127.0.0.1:53698)
0621 02:55:04.091784 (+ 25868us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:04.091844 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:04.092036 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:04.101216 (+ 9180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:04.101225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:04.101256 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:04.101723 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.101741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.103484 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.103492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.105936 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.105954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.108111 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.108120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.108368 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.108416 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 02:55:04.108489 (+ 73us) client_negotiation.cc:240] Negotiation successful
0621 02:55:04.108644 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25560,"thread_start_us":27082,"threads_started":1}
I20260621 02:55:04.110153 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.066202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.11.109.3:45015)
0621 02:55:04.100618 (+ 34416us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.100629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.100662 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:55:04.100758 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.100765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.100977 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.101199 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:04.101977 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.103311 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.106161 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.107862 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.109420 (+ 1558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.109693 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.109783 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.109911 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34236,"thread_start_us":34077,"threads_started":1}
I20260621 02:55:04.114441 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.066644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36225 (local address 127.0.0.1:48160)
0621 02:55:04.099914 (+ 33270us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:04.099931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:04.100046 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:04.108076 (+ 8030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:04.108084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:04.108106 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:04.108832 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.108848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.110950 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.110958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.112288 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.112300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.113670 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.113676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.113847 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.113878 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:55:04.114002 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 02:55:04.114126 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33137,"thread_start_us":25588,"threads_started":1}
I20260621 02:55:04.114542 30427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.067275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.11.109.2:36225)
0621 02:55:04.090956 (+ 23681us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.090970 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.100027 (+ 9057us) server_negotiation.cc:407] Connection header received
0621 02:55:04.100200 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.100210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.100489 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.100680 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:04.109028 (+ 8348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.110733 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.112450 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.113464 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.113613 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.113803 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.114054 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.114206 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23443,"thread_start_us":23698,"threads_started":1}
I20260621 02:55:04.113979 30078 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf6278b6274e718dce4d521461cb56 (DEFAULT_TABLE table=test-table [id=adb7f8b859fc4dd79b35ac1981a27244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:04.113979 30345 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf6278b6274e718dce4d521461cb56 (DEFAULT_TABLE table=test-table [id=adb7f8b859fc4dd79b35ac1981a27244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:04.115545 30345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf6278b6274e718dce4d521461cb56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:04.115543 30078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf6278b6274e718dce4d521461cb56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:04.118059 30212 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf6278b6274e718dce4d521461cb56 (DEFAULT_TABLE table=test-table [id=adb7f8b859fc4dd79b35ac1981a27244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:04.119117 30212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf6278b6274e718dce4d521461cb56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:04.132510 30430 tablet_bootstrap.cc:492] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Bootstrap starting.
I20260621 02:55:04.134713 30428 tablet_bootstrap.cc:492] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Bootstrap starting.
I20260621 02:55:04.136009 30429 tablet_bootstrap.cc:492] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Bootstrap starting.
I20260621 02:55:04.136510 30430 tablet_bootstrap.cc:654] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:04.137898 30430 log.cc:824] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:04.138285 30428 tablet_bootstrap.cc:654] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:04.139315 30428 log.cc:824] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:04.139786 30429 tablet_bootstrap.cc:654] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:04.141275 30429 log.cc:824] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:04.141855 30428 tablet_bootstrap.cc:492] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: No bootstrap required, opened a new log
I20260621 02:55:04.142189 30428 ts_tablet_manager.cc:1397] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260621 02:55:04.144526 30429 tablet_bootstrap.cc:492] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: No bootstrap required, opened a new log
I20260621 02:55:04.144797 30429 ts_tablet_manager.cc:1397] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 02:55:04.147563 30430 tablet_bootstrap.cc:492] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: No bootstrap required, opened a new log
I20260621 02:55:04.147814 30430 ts_tablet_manager.cc:1397] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260621 02:55:04.151247 30428 raft_consensus.cc:348] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.151675 30428 raft_consensus.cc:374] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:04.151794 30428 raft_consensus.cc:729] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde213152bf44f37bbe17d3330bda849, State: Initialized, Role: FOLLOWER
I20260621 02:55:04.152114 30429 raft_consensus.cc:348] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.152484 30429 raft_consensus.cc:374] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:04.152393 30428 consensus_queue.cc:260] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.152613 30429 raft_consensus.cc:729] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88beb88434ff4bc1991d3b21f522bda4, State: Initialized, Role: FOLLOWER
I20260621 02:55:04.153185 30429 consensus_queue.cc:260] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.153574 30277 heartbeater.cc:503] Master 127.11.109.62:33401 was elected leader, sending a full tablet report...
I20260621 02:55:04.154878 30429 ts_tablet_manager.cc:1428] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260621 02:55:04.156849 30428 ts_tablet_manager.cc:1428] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
I20260621 02:55:04.157881 30430 raft_consensus.cc:348] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.158303 30430 raft_consensus.cc:374] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:04.158406 30430 raft_consensus.cc:729] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d85b6a76bb43ac96cdec9f4db9f359, State: Initialized, Role: FOLLOWER
I20260621 02:55:04.158955 30430 consensus_queue.cc:260] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.159926 30410 heartbeater.cc:503] Master 127.11.109.62:33401 was elected leader, sending a full tablet report...
I20260621 02:55:04.160529 30430 ts_tablet_manager.cc:1428] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260621 02:55:04.240667 30144 tablet.cc:2369] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:55:04.241914 30278 tablet.cc:2369] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:55:04.244007 30411 tablet.cc:2369] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:55:04.271585 30436 raft_consensus.cc:482] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:55:04.271855 30436 raft_consensus.cc:504] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.273590 30436 leader_election.cc:288] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88beb88434ff4bc1991d3b21f522bda4 (127.11.109.1:36403), dde213152bf44f37bbe17d3330bda849 (127.11.109.2:36225)
I20260621 02:55:04.281133 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.273944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:51703 (local address 127.11.109.1:36403)
0621 02:55:04.274156 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.274163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.274762 (+ 599us) server_negotiation.cc:407] Connection header received
0621 02:55:04.274833 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.274836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.274898 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.275028 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:04.275487 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.276540 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.278951 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.280143 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.280344 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.280430 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.280750 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.280894 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:04.281150 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.274078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:36403 (local address 127.11.109.3:51703)
0621 02:55:04.274455 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:04.274482 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:04.274646 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:04.275080 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:04.275085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:04.275115 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:04.275325 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.275337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.276688 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.276693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.278785 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.278800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.280322 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.280326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.280555 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.280598 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:55:04.280690 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:55:04.280829 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":4}
I20260621 02:55:04.281219 30427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.274312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:41245 (local address 127.11.109.2:36225)
0621 02:55:04.274573 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.274581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.274964 (+ 383us) server_negotiation.cc:407] Connection header received
0621 02:55:04.275074 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.275081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.275155 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.275242 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:04.275826 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.276803 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.279101 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.280325 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.280470 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.280528 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.280827 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.280942 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260621 02:55:04.281258 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.274103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:36225 (local address 127.11.109.3:41245)
0621 02:55:04.274854 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:04.274867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:04.274961 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:04.275291 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:04.275296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:04.275316 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:04.275654 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.275662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.276949 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.276953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.278971 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:04.278980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.280511 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.280515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:04.280672 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.280714 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 02:55:04.280785 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 02:55:04.280873 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":2,"thread_start_us":384,"threads_started":1}
I20260621 02:55:04.282519 30232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf6278b6274e718dce4d521461cb56" candidate_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde213152bf44f37bbe17d3330bda849" is_pre_election: true
I20260621 02:55:04.282594 30098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf6278b6274e718dce4d521461cb56" candidate_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88beb88434ff4bc1991d3b21f522bda4" is_pre_election: true
I20260621 02:55:04.283113 30098 raft_consensus.cc:2454] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d85b6a76bb43ac96cdec9f4db9f359 in term 0.
I20260621 02:55:04.283111 30232 raft_consensus.cc:2454] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d85b6a76bb43ac96cdec9f4db9f359 in term 0.
I20260621 02:55:04.283821 30300 leader_election.cc:302] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d85b6a76bb43ac96cdec9f4db9f359, dde213152bf44f37bbe17d3330bda849; no voters:
I20260621 02:55:04.284214 30436 raft_consensus.cc:2781] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:55:04.284356 30436 raft_consensus.cc:482] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:55:04.284473 30436 raft_consensus.cc:3037] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:04.286626 30436 raft_consensus.cc:504] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.287230 30436 leader_election.cc:288] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [CANDIDATE]: Term 1 election: Requested vote from peers 88beb88434ff4bc1991d3b21f522bda4 (127.11.109.1:36403), dde213152bf44f37bbe17d3330bda849 (127.11.109.2:36225)
I20260621 02:55:04.287606 30098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf6278b6274e718dce4d521461cb56" candidate_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88beb88434ff4bc1991d3b21f522bda4"
I20260621 02:55:04.287657 30232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf6278b6274e718dce4d521461cb56" candidate_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde213152bf44f37bbe17d3330bda849"
I20260621 02:55:04.287819 30098 raft_consensus.cc:3037] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:04.287840 30232 raft_consensus.cc:3037] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:04.291011 30232 raft_consensus.cc:2454] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d85b6a76bb43ac96cdec9f4db9f359 in term 1.
I20260621 02:55:04.291011 30098 raft_consensus.cc:2454] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d85b6a76bb43ac96cdec9f4db9f359 in term 1.
I20260621 02:55:04.291450 30300 leader_election.cc:302] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d85b6a76bb43ac96cdec9f4db9f359, dde213152bf44f37bbe17d3330bda849; no voters:
I20260621 02:55:04.291745 30436 raft_consensus.cc:2781] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:04.292189 30436 raft_consensus.cc:686] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 1 LEADER]: Becoming Leader. State: Replica: 33d85b6a76bb43ac96cdec9f4db9f359, State: Running, Role: LEADER
I20260621 02:55:04.292691 30436 consensus_queue.cc:237] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } }
I20260621 02:55:04.297881 29954 catalog_manager.cc:5495] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d85b6a76bb43ac96cdec9f4db9f359 (127.11.109.3). New cstate: current_term: 1 leader_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d85b6a76bb43ac96cdec9f4db9f359" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 45015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 } health_report { overall_health: UNKNOWN } } }
I20260621 02:55:04.313560 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.308272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.11.109.3:45015)
0621 02:55:04.308465 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.308474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.308544 (+ 70us) server_negotiation.cc:407] Connection header received
0621 02:55:04.308716 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.308720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.308791 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.308942 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:04.309381 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.310070 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.311432 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.311814 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.311952 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.312490 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:04.312944 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:04.313140 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.313301 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.313345 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:55:04.320971 30098 raft_consensus.cc:1261] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 1 FOLLOWER]: Refusing update from remote peer 33d85b6a76bb43ac96cdec9f4db9f359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:04.321254 30232 raft_consensus.cc:1261] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 1 FOLLOWER]: Refusing update from remote peer 33d85b6a76bb43ac96cdec9f4db9f359: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:04.321996 30438 consensus_queue.cc:1035] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88beb88434ff4bc1991d3b21f522bda4" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:04.322438 30436 consensus_queue.cc:1035] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dde213152bf44f37bbe17d3330bda849" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:04.335675 30441 mvcc.cc:205] Tried to move back new op lower bound from 7299115025689448448 to 7299115025587081216. Current Snapshot: MvccSnapshot[applied={T|T < 7299115025689448448}]
I20260621 02:55:04.343024 30443 mvcc.cc:205] Tried to move back new op lower bound from 7299115025689448448 to 7299115025587081216. Current Snapshot: MvccSnapshot[applied={T|T < 7299115025689448448 or (T in {7299115025689448448})}]
I20260621 02:55:04.355090 29953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:04.355374 29953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:04.359256 29953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff50b2573248442a828647124d42a2e1: Sending DeleteTablet for 3 replicas of tablet 7dbf6278b6274e718dce4d521461cb56
I20260621 02:55:04.360327 30345 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbf6278b6274e718dce4d521461cb56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:04 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:53698
I20260621 02:55:04.360500 30212 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbf6278b6274e718dce4d521461cb56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:04 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:48160
I20260621 02:55:04.360494 30078 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbf6278b6274e718dce4d521461cb56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:04 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:45208
I20260621 02:55:04.364885 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30015
I20260621 02:55:04.373265 30452 tablet_replica.cc:330] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: stopping tablet replica
I20260621 02:55:04.373594 30451 tablet_replica.cc:330] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: stopping tablet replica
I20260621 02:55:04.373793 30453 tablet_replica.cc:330] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: stopping tablet replica
I20260621 02:55:04.373760 30452 raft_consensus.cc:2229] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:04.374212 30453 raft_consensus.cc:2229] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:04.374338 30452 raft_consensus.cc:2258] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:04.374715 30453 raft_consensus.cc:2258] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:04.375582 30451 raft_consensus.cc:2229] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:55:04.376214 30451 raft_consensus.cc:2258] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:04.376569 30452 ts_tablet_manager.cc:1905] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:04.377238 30453 ts_tablet_manager.cc:1905] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:04.384331 30451 ts_tablet_manager.cc:1905] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:04.384960 30453 ts_tablet_manager.cc:1918] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:04.385156 30453 log.cc:1196] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/7dbf6278b6274e718dce4d521461cb56
I20260621 02:55:04.385747 30453 ts_tablet_manager.cc:1939] T 7dbf6278b6274e718dce4d521461cb56 P 88beb88434ff4bc1991d3b21f522bda4: Deleting consensus metadata
I20260621 02:55:04.387640 29941 catalog_manager.cc:4841] TS 88beb88434ff4bc1991d3b21f522bda4 (127.11.109.1:36403): tablet 7dbf6278b6274e718dce4d521461cb56 (table test-table [id=adb7f8b859fc4dd79b35ac1981a27244]) successfully deleted
I20260621 02:55:04.391269 30451 ts_tablet_manager.cc:1918] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 21 02:55:04 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:55:04.392621 30452 ts_tablet_manager.cc:1918] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:04.392881 30452 log.cc:1196] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/7dbf6278b6274e718dce4d521461cb56
I20260621 02:55:04.393464 30452 ts_tablet_manager.cc:1939] T 7dbf6278b6274e718dce4d521461cb56 P dde213152bf44f37bbe17d3330bda849: Deleting consensus metadata
I20260621 02:55:04.395275 29940 catalog_manager.cc:4841] TS dde213152bf44f37bbe17d3330bda849 (127.11.109.2:36225): tablet 7dbf6278b6274e718dce4d521461cb56 (table test-table [id=adb7f8b859fc4dd79b35ac1981a27244]) successfully deleted
I20260621 02:55:04.395632 30451 log.cc:1196] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/7dbf6278b6274e718dce4d521461cb56
I20260621 02:55:04.396306 30451 ts_tablet_manager.cc:1939] T 7dbf6278b6274e718dce4d521461cb56 P 33d85b6a76bb43ac96cdec9f4db9f359: Deleting consensus metadata
I20260621 02:55:04.397986 29939 catalog_manager.cc:4841] TS 33d85b6a76bb43ac96cdec9f4db9f359 (127.11.109.3:45015): tablet 7dbf6278b6274e718dce4d521461cb56 (table test-table [id=adb7f8b859fc4dd79b35ac1981a27244]) successfully deleted
I20260621 02:55:04.400451 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.366137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.11.109.1:36403)
0621 02:55:04.366305 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.366311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.379989 (+ 13678us) server_negotiation.cc:407] Connection header received
0621 02:55:04.380465 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.380471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.380547 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.380618 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:04.382655 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.383885 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.386027 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.386639 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.386751 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.392600 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:04.392614 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:04.392632 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:04.392670 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:04.396515 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.397519 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.397529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.397532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.398175 (+ 643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.398634 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.398641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.398647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.399066 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:04.399352 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.399690 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.399907 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:55:04.401119 30138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:04.526444 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30015
I20260621 02:55:04.545363 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30149
Jun 21 02:55:04 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:55:04.563192 30427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.546207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.11.109.2:36225)
0621 02:55:04.546379 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.546383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.546419 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:55:04.546624 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.546628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.546691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.546802 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:04.548490 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.549504 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.551424 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.551804 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.551911 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.557465 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:04.557481 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:04.557498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:04.557540 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:04.560331 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.561038 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.561046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.561048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.561164 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.561581 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.561587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.561590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.561992 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:04.562268 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.562603 (+ 335us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.562814 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:55:04.563813 30272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:04.676858 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30149
I20260621 02:55:04.693495 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30282
Jun 21 02:55:04 dist-test-slave-bc52 krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:55:04.709578 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:04.694269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.11.109.3:45015)
0621 02:55:04.694480 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:04.694488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:04.694509 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:55:04.694713 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:04.694722 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:04.694798 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:04.694929 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:04.696136 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.696963 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.698277 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:04.698736 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:04.698870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:04.703391 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:04.703401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:04.703416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:04.703446 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:04.706754 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.707444 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.707451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.707457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.707604 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:04.707982 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:04.707990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:04.707993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:04.708360 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:04.708699 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:04.708966 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 02:55:04.709206 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:55:04.710379 30405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:04.828689 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30282
I20260621 02:55:04.847970 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 29922
I20260621 02:55:04.848986 29984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:04.957682 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 29922
2026-06-21T02:55:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4328 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T02:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:05Z Disabled control of system clock
I20260621 02:55:05.032840 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45601
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:55:05.351212 30462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:05.351550 30462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:05.351720 30462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:05.358729 30462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:05.358822 30462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:05.358912 30462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:05.359021 30462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:05.359083 30462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:05.367899 30462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:45601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:05.369140 30462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:05.370811 30462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:05.381831 30470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:05.382330 30468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:05.382480 30467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:05.383256 30462 server_base.cc:1034] running on GCE node
I20260621 02:55:05.384001 30462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:05.385394 30462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:05.386612 30462 hybrid_clock.cc:648] HybridClock initialized: now 1782010505386565 us; error 56 us; skew 500 ppm
I20260621 02:55:05.386991 30462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:05.389112 30462 webserver.cc:460] Webserver started at http://127.11.109.62:38161/ using document root <none> and password file <none>
I20260621 02:55:05.389775 30462 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:05.389900 30462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:05.390202 30462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:05.393044 30462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "6c201f9dbbe941bf94a5d7e88b5760ea"
format_stamp: "Formatted at 2026-06-21 02:55:05 on dist-test-slave-bc52"
server_key: "acbca2c44b90f8859dbcc98a809c3575"
server_key_iv: "526bea4148a474e0079aa9ea61d6895f"
server_key_version: "encryptionkey@0"
I20260621 02:55:05.393874 30462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "6c201f9dbbe941bf94a5d7e88b5760ea"
format_stamp: "Formatted at 2026-06-21 02:55:05 on dist-test-slave-bc52"
server_key: "acbca2c44b90f8859dbcc98a809c3575"
server_key_iv: "526bea4148a474e0079aa9ea61d6895f"
server_key_version: "encryptionkey@0"
I20260621 02:55:05.398622 30462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:55:05.402451 30476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:05.403959 30462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:55:05.404147 30462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "6c201f9dbbe941bf94a5d7e88b5760ea"
format_stamp: "Formatted at 2026-06-21 02:55:05 on dist-test-slave-bc52"
server_key: "acbca2c44b90f8859dbcc98a809c3575"
server_key_iv: "526bea4148a474e0079aa9ea61d6895f"
server_key_version: "encryptionkey@0"
I20260621 02:55:05.404361 30462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:05.418293 30462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:05.419093 30462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:05.419395 30462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:05.437055 30462 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:45601
I20260621 02:55:05.437075 30527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:45601 every 8 connection(s)
I20260621 02:55:05.438383 30462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:05.442080 30528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:05.443485 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30462
I20260621 02:55:05.443655 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:05.444123 11700 external_mini_cluster.cc:1428] Setting key 869688ee61bad2afb796e3a0aab61f5f
I20260621 02:55:05.456655 30528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea: Bootstrap starting.
I20260621 02:55:05.458637 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:05.446028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.11.109.62:45601)
0621 02:55:05.447106 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:05.447121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:05.447168 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:55:05.448848 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:05.448893 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:05.449528 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:05.449978 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:05.450464 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:05.452265 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:05.453664 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:05.454708 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:05.454881 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:05.455963 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:05.455972 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:05.455988 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:05.456675 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:05.456774 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:05.456808 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:05.456923 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:05.457082 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 02:55:05.457389 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":333,"threads_started":1}
I20260621 02:55:05.460125 30528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:05.461184 30528 log.cc:824] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:05.463661 30528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea: No bootstrap required, opened a new log
I20260621 02:55:05.471364 30528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45601 } }
I20260621 02:55:05.471694 30528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:05.471786 30528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c201f9dbbe941bf94a5d7e88b5760ea, State: Initialized, Role: FOLLOWER
I20260621 02:55:05.472329 30528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45601 } }
I20260621 02:55:05.472558 30528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:05.472687 30528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:05.472864 30528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:05.475104 30528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45601 } }
I20260621 02:55:05.475584 30528 leader_election.cc:302] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c201f9dbbe941bf94a5d7e88b5760ea; no voters:
I20260621 02:55:05.476143 30528 leader_election.cc:288] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:05.476294 30533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:05.477061 30533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [term 1 LEADER]: Becoming Leader. State: Replica: 6c201f9dbbe941bf94a5d7e88b5760ea, State: Running, Role: LEADER
I20260621 02:55:05.477589 30533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45601 } }
I20260621 02:55:05.478268 30528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:05.480360 30535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c201f9dbbe941bf94a5d7e88b5760ea. Latest consensus state: current_term: 1 leader_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45601 } } }
I20260621 02:55:05.480792 30535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:05.480683 30534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c201f9dbbe941bf94a5d7e88b5760ea" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 45601 } } }
I20260621 02:55:05.480940 30534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:05.482550 30541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:05.487963 30541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:05.498023 30541 catalog_manager.cc:1347] Generated new cluster ID: 4007a255862244a3a148a7bb60433334
I20260621 02:55:05.498121 30541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:05.514222 30541 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:05.515913 30541 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:05.532294 30541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea: Generated new TSK 0
I20260621 02:55:05.533120 30541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:05.541415 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45601
--builtin_ntp_servers=127.11.109.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:55:05.873833 30552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:05.874079 30552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:05.874231 30552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:05.881428 30552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:05.881511 30552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:05.881699 30552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:55:05.890218 30552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:05.891371 30552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:05.893069 30552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:05.905122 30557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:05.905092 30558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:05.905215 30560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:05.905782 30552 server_base.cc:1034] running on GCE node
I20260621 02:55:05.906456 30552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:05.907392 30552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:05.908600 30552 hybrid_clock.cc:648] HybridClock initialized: now 1782010505908544 us; error 52 us; skew 500 ppm
I20260621 02:55:05.908900 30552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:05.910888 30552 webserver.cc:460] Webserver started at http://127.11.109.1:33315/ using document root <none> and password file <none>
I20260621 02:55:05.911531 30552 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:05.911653 30552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:05.912006 30552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:05.914745 30552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "cab9fad948104db18e170a3e456b1dac"
format_stamp: "Formatted at 2026-06-21 02:55:05 on dist-test-slave-bc52"
server_key: "d45e1da8073f4e5da56aa14d8d2f682c"
server_key_iv: "910d587aed01ffe2a21b104fae77fe4c"
server_key_version: "encryptionkey@0"
I20260621 02:55:05.915500 30552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "cab9fad948104db18e170a3e456b1dac"
format_stamp: "Formatted at 2026-06-21 02:55:05 on dist-test-slave-bc52"
server_key: "d45e1da8073f4e5da56aa14d8d2f682c"
server_key_iv: "910d587aed01ffe2a21b104fae77fe4c"
server_key_version: "encryptionkey@0"
I20260621 02:55:05.920233 30552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260621 02:55:05.923354 30566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:05.924506 30552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 02:55:05.924660 30552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "cab9fad948104db18e170a3e456b1dac"
format_stamp: "Formatted at 2026-06-21 02:55:05 on dist-test-slave-bc52"
server_key: "d45e1da8073f4e5da56aa14d8d2f682c"
server_key_iv: "910d587aed01ffe2a21b104fae77fe4c"
server_key_version: "encryptionkey@0"
I20260621 02:55:05.924898 30552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:05.953886 30552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:05.954624 30552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:05.954914 30552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:05.955814 30552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:05.957273 30552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:05.957382 30552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:05.957504 30552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:05.957585 30552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:05.989710 30552 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:39491
I20260621 02:55:05.989816 30678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:39491 every 8 connection(s)
I20260621 02:55:05.990980 30552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:55:05.997979 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30552
I20260621 02:55:05.998153 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:55:05.998611 11700 external_mini_cluster.cc:1428] Setting key fe7437822d1564778f408b67a7054206
I20260621 02:55:06.001315 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45601
--builtin_ntp_servers=127.11.109.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:55:06.003746 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:05.993398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:56209 (local address 127.11.109.62:45601)
0621 02:55:05.993813 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:05.993817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:05.994860 (+ 1043us) server_negotiation.cc:407] Connection header received
0621 02:55:05.996728 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:05.996734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:05.996819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:05.996958 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:05.998265 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:05.999029 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.000871 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.001342 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.001495 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.003049 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:06.003054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:06.003057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:06.003075 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:06.003110 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:06.003121 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:06.003189 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.003433 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.003502 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":229,"threads_started":1}
I20260621 02:55:06.005143 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:05.993879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45601 (local address 127.11.109.1:56209)
0621 02:55:05.994710 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:05.994748 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:05.996479 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:05.997215 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:05.997232 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:05.997315 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:05.998047 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:05.998076 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:05.999191 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:05.999201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.000701 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.000713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.001587 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.001596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.001930 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.002692 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:55:06.002698 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:06.002753 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:55:06.002860 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:06.003219 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:06.003232 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:06.003242 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.003417 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.003811 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":266,"threads_started":1}
I20260621 02:55:06.006628 30679 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45601
I20260621 02:55:06.006942 30679 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:06.007839 30679 heartbeater.cc:511] Master 127.11.109.62:45601 requested a full tablet report, sending...
I20260621 02:55:06.010175 30493 ts_manager.cc:194] Registered new tserver with Master: cab9fad948104db18e170a3e456b1dac (127.11.109.1:39491)
I20260621 02:55:06.011621 30493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.1:56209
W20260621 02:55:06.319466 30683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:06.319705 30683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:06.319819 30683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:06.327102 30683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:06.327226 30683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:06.327404 30683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:55:06.336325 30683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:06.337584 30683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:06.339289 30683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:06.352253 30688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:06.352293 30691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:06.352329 30689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:06.352975 30683 server_base.cc:1034] running on GCE node
I20260621 02:55:06.353621 30683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:06.354529 30683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:06.355720 30683 hybrid_clock.cc:648] HybridClock initialized: now 1782010506355685 us; error 32 us; skew 500 ppm
I20260621 02:55:06.356045 30683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:06.358186 30683 webserver.cc:460] Webserver started at http://127.11.109.2:43603/ using document root <none> and password file <none>
I20260621 02:55:06.358829 30683 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:06.358939 30683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:06.359253 30683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:06.361996 30683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "8e45eead812340318461ea0d1d503e5c"
format_stamp: "Formatted at 2026-06-21 02:55:06 on dist-test-slave-bc52"
server_key: "0872239cebff53370b1cf9f68f648364"
server_key_iv: "2261a6858903f4d8be5c62ee05bacb57"
server_key_version: "encryptionkey@0"
I20260621 02:55:06.362799 30683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "8e45eead812340318461ea0d1d503e5c"
format_stamp: "Formatted at 2026-06-21 02:55:06 on dist-test-slave-bc52"
server_key: "0872239cebff53370b1cf9f68f648364"
server_key_iv: "2261a6858903f4d8be5c62ee05bacb57"
server_key_version: "encryptionkey@0"
I20260621 02:55:06.367925 30683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:55:06.371117 30697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:06.372324 30683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:55:06.372484 30683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "8e45eead812340318461ea0d1d503e5c"
format_stamp: "Formatted at 2026-06-21 02:55:06 on dist-test-slave-bc52"
server_key: "0872239cebff53370b1cf9f68f648364"
server_key_iv: "2261a6858903f4d8be5c62ee05bacb57"
server_key_version: "encryptionkey@0"
I20260621 02:55:06.372686 30683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:06.388706 30683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:06.389634 30683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:06.389932 30683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:06.390879 30683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:06.392382 30683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:06.392513 30683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:06.392637 30683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:06.392729 30683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:06.425397 30683 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:42755
I20260621 02:55:06.425484 30809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:42755 every 8 connection(s)
I20260621 02:55:06.426811 30683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:55:06.436595 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30683
I20260621 02:55:06.436784 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:55:06.437390 11700 external_mini_cluster.cc:1428] Setting key 225809b6c1d5791d2136d3dca54ea94e
I20260621 02:55:06.439222 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.429387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:53415 (local address 127.11.109.62:45601)
0621 02:55:06.429604 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.429610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.431029 (+ 1419us) server_negotiation.cc:407] Connection header received
0621 02:55:06.432693 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.432697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.432770 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.432857 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:06.433906 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.434643 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.436116 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.436597 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.436727 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.438350 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:06.438355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:06.438361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:06.438380 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:06.438409 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:06.438421 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:06.438540 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.438810 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.438897 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:55:06.440258 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45601
--builtin_ntp_servers=127.11.109.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:55:06.440490 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.430023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45601 (local address 127.11.109.2:53415)
0621 02:55:06.430872 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.430913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.432441 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.433056 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.433064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.433146 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:06.433754 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.433770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.434759 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.434762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.435971 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.435983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.436816 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.436824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.437085 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.437920 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:55:06.437925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:06.437976 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:55:06.438071 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:06.438557 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:06.438568 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:06.438576 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.438750 (+ 174us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.439288 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":266,"threads_started":1}
I20260621 02:55:06.442063 30810 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45601
I20260621 02:55:06.442364 30810 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:06.443197 30810 heartbeater.cc:511] Master 127.11.109.62:45601 requested a full tablet report, sending...
I20260621 02:55:06.444662 30493 ts_manager.cc:194] Registered new tserver with Master: 8e45eead812340318461ea0d1d503e5c (127.11.109.2:42755)
I20260621 02:55:06.445978 30493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.2:53415
W20260621 02:55:06.759426 30813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:06.759692 30813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:06.759773 30813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:06.766954 30813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:06.767078 30813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:06.767182 30813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:55:06.775956 30813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:06.777273 30813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:06.778972 30813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:06.791699 30818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:06.791651 30819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:06.791740 30821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:06.792570 30813 server_base.cc:1034] running on GCE node
I20260621 02:55:06.793248 30813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:06.795284 30813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:06.796521 30813 hybrid_clock.cc:648] HybridClock initialized: now 1782010506796473 us; error 45 us; skew 500 ppm
I20260621 02:55:06.796847 30813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:06.799151 30813 webserver.cc:460] Webserver started at http://127.11.109.3:43755/ using document root <none> and password file <none>
I20260621 02:55:06.799891 30813 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:06.800019 30813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:06.800359 30813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:06.803179 30813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "6c95e24c355b4ebaa419c00440edfade"
format_stamp: "Formatted at 2026-06-21 02:55:06 on dist-test-slave-bc52"
server_key: "e2f0b3a4ef8afeb9330fe2fafbd63fd1"
server_key_iv: "29c35e1ace4b522dbf7f7cb4b09ff667"
server_key_version: "encryptionkey@0"
I20260621 02:55:06.804018 30813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "6c95e24c355b4ebaa419c00440edfade"
format_stamp: "Formatted at 2026-06-21 02:55:06 on dist-test-slave-bc52"
server_key: "e2f0b3a4ef8afeb9330fe2fafbd63fd1"
server_key_iv: "29c35e1ace4b522dbf7f7cb4b09ff667"
server_key_version: "encryptionkey@0"
I20260621 02:55:06.808893 30813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:55:06.812234 30827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:06.813628 30813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:55:06.813789 30813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "6c95e24c355b4ebaa419c00440edfade"
format_stamp: "Formatted at 2026-06-21 02:55:06 on dist-test-slave-bc52"
server_key: "e2f0b3a4ef8afeb9330fe2fafbd63fd1"
server_key_iv: "29c35e1ace4b522dbf7f7cb4b09ff667"
server_key_version: "encryptionkey@0"
I20260621 02:55:06.813989 30813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:06.832096 30813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:06.832850 30813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:06.833148 30813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:06.834131 30813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:06.835561 30813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:06.835680 30813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:06.835817 30813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:06.835914 30813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:06.868126 30813 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:41725
I20260621 02:55:06.868180 30939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:41725 every 8 connection(s)
I20260621 02:55:06.869522 30813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:55:06.874683 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 30813
I20260621 02:55:06.874866 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:55:06.875299 11700 external_mini_cluster.cc:1428] Setting key c8da998ec5a0d4931925c8d0d1fc15fb
I20260621 02:55:06.882530 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.872091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:44963 (local address 127.11.109.62:45601)
0621 02:55:06.872290 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.872295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.873798 (+ 1503us) server_negotiation.cc:407] Connection header received
0621 02:55:06.876069 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.876075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.876158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.876299 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:06.877667 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.878588 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.880144 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.880528 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.880619 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.881827 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:06.881833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:06.881836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:06.881852 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:06.881883 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:06.881895 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:06.881987 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.882200 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.882304 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:06.883543 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.872635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:45601 (local address 127.11.109.3:44963)
0621 02:55:06.873669 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.873712 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.875740 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.876644 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.876652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.876739 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:06.877482 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.877501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.878718 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.878722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.880027 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.880038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.880700 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.880704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.880888 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.881530 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:55:06.881533 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:06.881569 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:55:06.881664 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:06.881993 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:06.882000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:06.882006 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.882178 (+ 172us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.882486 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":289,"threads_started":1}
I20260621 02:55:06.884759 30940 heartbeater.cc:347] Connected to a master server at 127.11.109.62:45601
I20260621 02:55:06.885035 30940 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:06.885761 30940 heartbeater.cc:511] Master 127.11.109.62:45601 requested a full tablet report, sending...
I20260621 02:55:06.887044 30493 ts_manager.cc:194] Registered new tserver with Master: 6c95e24c355b4ebaa419c00440edfade (127.11.109.3:41725)
I20260621 02:55:06.887754 30493 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.3:44963
I20260621 02:55:06.890791 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:55:06.900362 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.894583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.11.109.62:45601)
0621 02:55:06.894780 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.894788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.895144 (+ 356us) server_negotiation.cc:407] Connection header received
0621 02:55:06.895347 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.895351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.895411 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.895509 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:06.895973 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.896665 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.898040 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.898461 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.898592 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.899657 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:06.899661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:06.899665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:06.899681 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:06.899709 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:06.899721 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:06.899878 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.900060 (+ 182us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.900114 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:55:06.903993 30493 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:55:06.910671 30493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:55:06.933138 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.922136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.11.109.3:41725)
0621 02:55:06.923482 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.923491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.924029 (+ 538us) server_negotiation.cc:407] Connection header received
0621 02:55:06.924405 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.924416 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.924670 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.924848 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:06.925957 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.927495 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.930152 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.931561 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.931666 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.931926 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.932575 (+ 649us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.932749 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":995,"threads_started":1}
I20260621 02:55:06.934170 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.922018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:41725 (local address 127.0.0.1:42032)
0621 02:55:06.923879 (+ 1861us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.923925 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.924171 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.925219 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.925227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.925261 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:06.925717 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.925730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.927671 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.927676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.929959 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.929973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.932099 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.932106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.932331 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.932384 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.933629 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.933810 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"spinlock_wait_cycles":5376,"thread_start_us":1421,"threads_started":1}
I20260621 02:55:06.935774 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.922050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:42755 (local address 127.0.0.1:41190)
0621 02:55:06.923879 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.924155 (+ 276us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.924254 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.926482 (+ 2228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.926487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.926513 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:06.926873 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.926884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.930130 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.930135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.932171 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.932184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.935069 (+ 2885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.935075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.935250 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.935281 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.935346 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.935473 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":1599,"threads_started":1}
I20260621 02:55:06.937125 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.922757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.11.109.2:42755)
0621 02:55:06.924746 (+ 1989us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.924753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.924784 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:55:06.924879 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.924886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.926259 (+ 1373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.927067 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:06.927289 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.929980 (+ 2691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.932415 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.934862 (+ 2447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.936284 (+ 1422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.936562 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.936632 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.936774 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1818,"thread_start_us":1705,"threads_started":1}
I20260621 02:55:06.938439 30875 tablet_service.cc:1467] Processing CreateTablet for tablet ae9d1582bd0c4fa68e9df679b83ae307 (DEFAULT_TABLE table=test-table [id=66660557d7fd4fd39df79ba843522976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:06.939951 30875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9d1582bd0c4fa68e9df679b83ae307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:06.940551 30745 tablet_service.cc:1467] Processing CreateTablet for tablet ae9d1582bd0c4fa68e9df679b83ae307 (DEFAULT_TABLE table=test-table [id=66660557d7fd4fd39df79ba843522976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:06.941815 30745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9d1582bd0c4fa68e9df679b83ae307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:06.945328 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.924780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49768)
0621 02:55:06.935992 (+ 11212us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.936023 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.936128 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.939075 (+ 2947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.939080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.939107 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:06.939257 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.939269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.941023 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.941028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.942803 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.942820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.944558 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.944564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.944754 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.944806 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.944906 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.945035 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11105,"thread_start_us":12848,"threads_started":1}
I20260621 02:55:06.945729 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.925117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.11.109.1:39491)
0621 02:55:06.938174 (+ 13057us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.938183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.938211 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:55:06.938302 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.938308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.938522 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.939070 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:06.939408 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.940877 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.942968 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.944338 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.944517 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.944736 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.944964 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.945123 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12894,"thread_start_us":13493,"threads_started":1}
I20260621 02:55:06.949026 30614 tablet_service.cc:1467] Processing CreateTablet for tablet ae9d1582bd0c4fa68e9df679b83ae307 (DEFAULT_TABLE table=test-table [id=66660557d7fd4fd39df79ba843522976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:06.950275 30614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9d1582bd0c4fa68e9df679b83ae307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:06.953447 30958 tablet_bootstrap.cc:492] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Bootstrap starting.
I20260621 02:55:06.955355 30959 tablet_bootstrap.cc:492] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Bootstrap starting.
I20260621 02:55:06.957003 30958 tablet_bootstrap.cc:654] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:06.958247 30958 log.cc:824] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:06.958832 30959 tablet_bootstrap.cc:654] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:06.960165 30959 log.cc:824] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:06.961283 30958 tablet_bootstrap.cc:492] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: No bootstrap required, opened a new log
I20260621 02:55:06.961537 30958 ts_tablet_manager.cc:1397] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260621 02:55:06.962750 30959 tablet_bootstrap.cc:492] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: No bootstrap required, opened a new log
I20260621 02:55:06.963060 30959 ts_tablet_manager.cc:1397] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:55:06.963541 30962 tablet_bootstrap.cc:492] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Bootstrap starting.
I20260621 02:55:06.967271 30962 tablet_bootstrap.cc:654] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:06.968426 30962 log.cc:824] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:06.969029 30958 raft_consensus.cc:348] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.969514 30958 raft_consensus.cc:374] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:06.969626 30958 raft_consensus.cc:729] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e45eead812340318461ea0d1d503e5c, State: Initialized, Role: FOLLOWER
I20260621 02:55:06.970450 30958 consensus_queue.cc:260] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.971536 30962 tablet_bootstrap.cc:492] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: No bootstrap required, opened a new log
I20260621 02:55:06.971840 30962 ts_tablet_manager.cc:1397] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260621 02:55:06.972719 30959 raft_consensus.cc:348] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.973054 30959 raft_consensus.cc:374] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:06.973174 30959 raft_consensus.cc:729] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c95e24c355b4ebaa419c00440edfade, State: Initialized, Role: FOLLOWER
I20260621 02:55:06.973790 30959 consensus_queue.cc:260] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.974830 30940 heartbeater.cc:503] Master 127.11.109.62:45601 was elected leader, sending a full tablet report...
I20260621 02:55:06.975492 30959 ts_tablet_manager.cc:1428] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Time spent starting tablet: real 0.012s user 0.002s sys 0.009s
I20260621 02:55:06.978188 30958 ts_tablet_manager.cc:1428] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260621 02:55:06.978675 30810 heartbeater.cc:503] Master 127.11.109.62:45601 was elected leader, sending a full tablet report...
I20260621 02:55:06.982738 30965 raft_consensus.cc:482] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:55:06.983047 30965 raft_consensus.cc:504] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.983287 30962 raft_consensus.cc:348] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.983731 30962 raft_consensus.cc:374] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:06.983863 30962 raft_consensus.cc:729] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab9fad948104db18e170a3e456b1dac, State: Initialized, Role: FOLLOWER
I20260621 02:55:06.984447 30962 consensus_queue.cc:260] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.985049 30965 leader_election.cc:288] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e45eead812340318461ea0d1d503e5c (127.11.109.2:42755), cab9fad948104db18e170a3e456b1dac (127.11.109.1:39491)
I20260621 02:55:06.986140 30679 heartbeater.cc:503] Master 127.11.109.62:45601 was elected leader, sending a full tablet report...
I20260621 02:55:06.987179 30962 ts_tablet_manager.cc:1428] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260621 02:55:06.992429 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.986045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:40065 (local address 127.11.109.2:42755)
0621 02:55:06.986381 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.986387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.986404 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:55:06.986464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.986469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.986533 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.986621 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:06.987243 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.988298 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.990685 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.991569 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.991738 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.991806 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.992087 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.992203 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260621 02:55:06.992479 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.985517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:42755 (local address 127.11.109.3:40065)
0621 02:55:06.985806 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.985831 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.985966 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.986804 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.986808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.986834 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:06.987047 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.987060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.988431 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.988439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.990508 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.990522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.991728 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.991734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.991920 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.991959 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.992063 (+ 104us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.992196 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
W20260621 02:55:06.993247 30680 tablet.cc:2369] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:55:06.993585 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d1582bd0c4fa68e9df679b83ae307" candidate_uuid: "6c95e24c355b4ebaa419c00440edfade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e45eead812340318461ea0d1d503e5c" is_pre_election: true
I20260621 02:55:06.994076 30765 raft_consensus.cc:2454] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c95e24c355b4ebaa419c00440edfade in term 0.
I20260621 02:55:06.994767 30829 leader_election.cc:302] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c95e24c355b4ebaa419c00440edfade, 8e45eead812340318461ea0d1d503e5c; no voters:
I20260621 02:55:06.995126 30966 raft_consensus.cc:2781] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:55:06.995267 30966 raft_consensus.cc:482] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:55:06.995424 30966 raft_consensus.cc:3037] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:06.995802 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.987488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:45645 (local address 127.11.109.1:39491)
0621 02:55:06.987670 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:06.987675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:06.988181 (+ 506us) server_negotiation.cc:407] Connection header received
0621 02:55:06.988298 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:06.988305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:06.988404 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:06.988496 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:06.989389 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.990450 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.993542 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.994799 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.994910 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.994970 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:06.995444 (+ 474us) server_negotiation.cc:299] Negotiation successful
0621 02:55:06.995563 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:55:06.996424 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:06.986382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.11.109.3:45645)
0621 02:55:06.988065 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:06.988085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:06.988196 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:06.988649 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:06.988653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:06.988679 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:06.989143 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.989157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.990633 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.990641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.993298 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:06.993321 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:06.995100 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:06.995105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:06.995278 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:06.995307 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 02:55:06.995960 (+ 653us) client_negotiation.cc:240] Negotiation successful
0621 02:55:06.996114 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":1395,"threads_started":1}
I20260621 02:55:06.997265 30634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d1582bd0c4fa68e9df679b83ae307" candidate_uuid: "6c95e24c355b4ebaa419c00440edfade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab9fad948104db18e170a3e456b1dac" is_pre_election: true
I20260621 02:55:06.997749 30634 raft_consensus.cc:2454] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c95e24c355b4ebaa419c00440edfade in term 0.
I20260621 02:55:06.998046 30966 raft_consensus.cc:504] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:06.998840 30966 leader_election.cc:288] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [CANDIDATE]: Term 1 election: Requested vote from peers 8e45eead812340318461ea0d1d503e5c (127.11.109.2:42755), cab9fad948104db18e170a3e456b1dac (127.11.109.1:39491)
I20260621 02:55:06.999328 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d1582bd0c4fa68e9df679b83ae307" candidate_uuid: "6c95e24c355b4ebaa419c00440edfade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e45eead812340318461ea0d1d503e5c"
I20260621 02:55:06.999516 30765 raft_consensus.cc:3037] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:07.001394 30634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9d1582bd0c4fa68e9df679b83ae307" candidate_uuid: "6c95e24c355b4ebaa419c00440edfade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab9fad948104db18e170a3e456b1dac"
I20260621 02:55:07.001581 30634 raft_consensus.cc:3037] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:07.002573 30765 raft_consensus.cc:2454] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c95e24c355b4ebaa419c00440edfade in term 1.
I20260621 02:55:07.002971 30829 leader_election.cc:302] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c95e24c355b4ebaa419c00440edfade, 8e45eead812340318461ea0d1d503e5c; no voters:
I20260621 02:55:07.003343 30966 raft_consensus.cc:2781] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:07.003998 30634 raft_consensus.cc:2454] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c95e24c355b4ebaa419c00440edfade in term 1.
I20260621 02:55:07.003966 30966 raft_consensus.cc:686] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 1 LEADER]: Becoming Leader. State: Replica: 6c95e24c355b4ebaa419c00440edfade, State: Running, Role: LEADER
I20260621 02:55:07.004601 30966 consensus_queue.cc:237] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } }
I20260621 02:55:07.009603 30493 catalog_manager.cc:5495] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c95e24c355b4ebaa419c00440edfade (127.11.109.3). New cstate: current_term: 1 leader_uuid: "6c95e24c355b4ebaa419c00440edfade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c95e24c355b4ebaa419c00440edfade" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 41725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 } health_report { overall_health: UNKNOWN } } }
I20260621 02:55:07.029964 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.024886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.11.109.3:41725)
0621 02:55:07.025052 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:07.025056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:07.025167 (+ 111us) server_negotiation.cc:407] Connection header received
0621 02:55:07.025395 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:07.025398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:07.025472 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:07.025592 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:07.025904 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.026586 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.027891 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.028286 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.028420 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:07.028940 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:07.029469 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:07.029642 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:07.029744 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 02:55:07.029778 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:55:07.037484 30634 raft_consensus.cc:1261] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 1 FOLLOWER]: Refusing update from remote peer 6c95e24c355b4ebaa419c00440edfade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:07.037525 30765 raft_consensus.cc:1261] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 1 FOLLOWER]: Refusing update from remote peer 6c95e24c355b4ebaa419c00440edfade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:07.038369 30965 consensus_queue.cc:1035] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab9fad948104db18e170a3e456b1dac" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:07.038700 30966 consensus_queue.cc:1035] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e45eead812340318461ea0d1d503e5c" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:07.048329 30972 mvcc.cc:205] Tried to move back new op lower bound from 7299115036816609280 to 7299115036695379968. Current Snapshot: MvccSnapshot[applied={T|T < 7299115036816609280}]
I20260621 02:55:07.065721 30492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:07.066033 30492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:07.069648 30492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c201f9dbbe941bf94a5d7e88b5760ea: Sending DeleteTablet for 3 replicas of tablet ae9d1582bd0c4fa68e9df679b83ae307
I20260621 02:55:07.070675 30875 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9d1582bd0c4fa68e9df679b83ae307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:07 UTC) from {username='slave'} at 127.0.0.1:42032
I20260621 02:55:07.070823 30745 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9d1582bd0c4fa68e9df679b83ae307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:07 UTC) from {username='slave'} at 127.0.0.1:41190
I20260621 02:55:07.071314 30614 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9d1582bd0c4fa68e9df679b83ae307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:07 UTC) from {username='slave'} at 127.0.0.1:49768
I20260621 02:55:07.072392 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30552
I20260621 02:55:07.072450 30981 tablet_replica.cc:330] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: stopping tablet replica
I20260621 02:55:07.072846 30981 raft_consensus.cc:2229] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:55:07.073587 30981 raft_consensus.cc:2258] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:07.075739 30981 ts_tablet_manager.cc:1905] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:07.085078 30981 ts_tablet_manager.cc:1918] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:07.085377 30981 log.cc:1196] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/ae9d1582bd0c4fa68e9df679b83ae307
I20260621 02:55:07.086056 30981 ts_tablet_manager.cc:1939] T ae9d1582bd0c4fa68e9df679b83ae307 P 6c95e24c355b4ebaa419c00440edfade: Deleting consensus metadata
I20260621 02:55:07.088109 30480 catalog_manager.cc:4841] TS 6c95e24c355b4ebaa419c00440edfade (127.11.109.3:41725): tablet ae9d1582bd0c4fa68e9df679b83ae307 (table test-table [id=66660557d7fd4fd39df79ba843522976]) successfully deleted
I20260621 02:55:07.089637 30983 tablet_replica.cc:330] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: stopping tablet replica
I20260621 02:55:07.090299 30983 raft_consensus.cc:2229] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:07.090943 30983 raft_consensus.cc:2258] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:07.091905 30982 tablet_replica.cc:330] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: stopping tablet replica
I20260621 02:55:07.092358 30982 raft_consensus.cc:2229] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:07.092868 30982 raft_consensus.cc:2258] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:07.093446 30983 ts_tablet_manager.cc:1905] T ae9d1582bd0c4fa68e9df679b83ae307 P cab9fad948104db18e170a3e456b1dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:07.095074 30982 ts_tablet_manager.cc:1905] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:07.097478 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.075310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.11.109.1:39491)
0621 02:55:07.075491 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:07.075497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:07.088973 (+ 13476us) server_negotiation.cc:407] Connection header received
0621 02:55:07.089217 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:07.089225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:07.089354 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:07.089493 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:07.090251 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.091528 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.093498 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.094124 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.094229 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:07.095711 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:07.095721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:07.095737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:07.096504 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:07.096559 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:07.096589 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:07.096667 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:07.096905 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 02:55:07.097023 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:55:07.098805 30674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:07.102938 30982 ts_tablet_manager.cc:1918] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:07.103109 30982 log.cc:1196] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/ae9d1582bd0c4fa68e9df679b83ae307
I20260621 02:55:07.103609 30982 ts_tablet_manager.cc:1939] T ae9d1582bd0c4fa68e9df679b83ae307 P 8e45eead812340318461ea0d1d503e5c: Deleting consensus metadata
I20260621 02:55:07.105587 30478 catalog_manager.cc:4841] TS 8e45eead812340318461ea0d1d503e5c (127.11.109.2:42755): tablet ae9d1582bd0c4fa68e9df679b83ae307 (table test-table [id=66660557d7fd4fd39df79ba843522976]) successfully deleted
I20260621 02:55:07.188889 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30552
I20260621 02:55:07.211850 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30683
W20260621 02:55:07.211871 30479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.109.1:39491 (error 108)
I20260621 02:55:07.214036 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.213495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49778)
0621 02:55:07.213711 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.213846 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 02:55:07.214284 30479 catalog_manager.cc:4584] TS cab9fad948104db18e170a3e456b1dac (127.11.109.1:39491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae9d1582bd0c4fa68e9df679b83ae307: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
I20260621 02:55:07.219233 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.213048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.11.109.2:42755)
0621 02:55:07.213253 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:07.213257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:07.213274 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:55:07.213452 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:07.213458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:07.213562 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:07.213666 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:07.214086 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.214904 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.216156 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.216541 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.216695 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:07.217802 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:07.217811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:07.217825 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:07.218510 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:07.218555 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:07.218574 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:07.218690 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:07.218833 (+ 143us) server_negotiation.cc:299] Negotiation successful
0621 02:55:07.218940 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:55:07.219923 30805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:07.268046 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.267482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49786)
0621 02:55:07.267735 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.267853 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260621 02:55:07.269249 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.268806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49798)
0621 02:55:07.268971 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.269057 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:55:07.310072 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30683
I20260621 02:55:07.329025 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30813
I20260621 02:55:07.329917 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.329405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49802)
0621 02:55:07.329639 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.329738 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:55:07.331187 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.330775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49814)
0621 02:55:07.330945 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.331045 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:55:07.337515 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.329971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.11.109.3:41725)
0621 02:55:07.330148 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:07.330156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:07.330176 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:55:07.330346 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:07.330353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:07.330467 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:07.330594 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:07.331097 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.332344 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.333839 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:07.334523 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:07.334659 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:07.335724 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:07.335735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:07.335753 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:07.336548 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:07.336636 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:07.336662 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:07.336784 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:07.336989 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 02:55:07.337084 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:55:07.338263 30935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:07.413076 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.412530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49828)
0621 02:55:07.412771 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.412895 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260621 02:55:07.414253 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.413773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49836)
0621 02:55:07.413995 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.414093 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260621 02:55:07.466444 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30813
I20260621 02:55:07.486461 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 30462
I20260621 02:55:07.487566 30523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:07.594563 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:07.594174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:39491 (local address 127.0.0.1:49852)
0621 02:55:07.594336 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:07.594416 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 02:55:07.594614 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 30462
2026-06-21T02:55:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2636 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:07 dist-test-slave-bc52 krb5kdc[30992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:07 dist-test-slave-bc52 krb5kdc[30992](info): set up 2 sockets
Jun 21 02:55:07 dist-test-slave-bc52 krb5kdc[30992](info): commencing operation
krb5kdc: starting...
W20260621 02:55:08.685576 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:08 dist-test-slave-bc52 krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:08.924405 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41723
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:09.256453 31005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:09.256707 31005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:09.256870 31005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:09.263713 31005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:09.263798 31005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:09.263852 31005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:09.263926 31005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:09.263986 31005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:09.273839 31005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:09.275123 31005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:09.276412 31005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:09.287627 31010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:09.287662 31011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:09.287698 31013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:09.288697 31005 server_base.cc:1034] running on GCE node
I20260621 02:55:09.289464 31005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:09.290683 31005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:09.291905 31005 hybrid_clock.cc:648] HybridClock initialized: now 1782010509291863 us; error 51 us; skew 500 ppm
I20260621 02:55:09.292249 31005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:09 dist-test-slave-bc52 krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010509, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:09.295186 31005 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:09.296591 31005 webserver.cc:460] Webserver started at http://127.11.109.62:36143/ using document root <none> and password file <none>
I20260621 02:55:09.297236 31005 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:09.297346 31005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:09.297595 31005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:09.300437 31005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "6921e7cccf764e9a9cb7bb4248e19552"
format_stamp: "Formatted at 2026-06-21 02:55:09 on dist-test-slave-bc52"
server_key: "3a9c1fd015a8bcc5cff911ebcda774c9"
server_key_iv: "8376f872adfc2acf5e357cb83b7aeb24"
server_key_version: "encryptionkey@0"
I20260621 02:55:09.301115 31005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "6921e7cccf764e9a9cb7bb4248e19552"
format_stamp: "Formatted at 2026-06-21 02:55:09 on dist-test-slave-bc52"
server_key: "3a9c1fd015a8bcc5cff911ebcda774c9"
server_key_iv: "8376f872adfc2acf5e357cb83b7aeb24"
server_key_version: "encryptionkey@0"
I20260621 02:55:09.306262 31005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:55:09.309599 31020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:09.310942 31005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:55:09.311115 31005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "6921e7cccf764e9a9cb7bb4248e19552"
format_stamp: "Formatted at 2026-06-21 02:55:09 on dist-test-slave-bc52"
server_key: "3a9c1fd015a8bcc5cff911ebcda774c9"
server_key_iv: "8376f872adfc2acf5e357cb83b7aeb24"
server_key_version: "encryptionkey@0"
I20260621 02:55:09.311411 31005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:09.335394 31005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:09.339622 31005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:09.339900 31005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:09.358265 31005 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:41723
I20260621 02:55:09.358297 31071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:41723 every 8 connection(s)
I20260621 02:55:09.359603 31005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:09.363142 31072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:09.366307 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31005
I20260621 02:55:09.366487 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:09.366922 11700 external_mini_cluster.cc:1428] Setting key 10b635fa3f8296efe5d33bc1e78d5ee3
I20260621 02:55:09.375967 31072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552: Bootstrap starting.
I20260621 02:55:09.379501 31072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:09.380468 31072 log.cc:824] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:55:09 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:09.383255 31072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552: No bootstrap required, opened a new log
I20260621 02:55:09.390158 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:09.368762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.11.109.62:41723)
0621 02:55:09.369562 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:09.369576 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:09.369625 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:55:09.371813 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:09.371843 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:09.372365 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:09.372795 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:09.374152 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:09.376089 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:09.377443 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:09.378481 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:09.378687 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:09.382904 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:09.382912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:09.382926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:09.382963 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:09.386188 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:09.386936 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:09.386942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:09.386948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:09.387121 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:09.387541 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:09.387545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:09.387547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:09.388040 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:09.388368 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:09.388790 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 02:55:09.389164 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":342,"threads_started":1}
I20260621 02:55:09.391032 31072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6921e7cccf764e9a9cb7bb4248e19552" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41723 } }
I20260621 02:55:09.391355 31072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:09.391472 31072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6921e7cccf764e9a9cb7bb4248e19552, State: Initialized, Role: FOLLOWER
I20260621 02:55:09.392060 31072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6921e7cccf764e9a9cb7bb4248e19552" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41723 } }
I20260621 02:55:09.392290 31072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:09.392457 31072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:09.392627 31072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:09.394840 31072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6921e7cccf764e9a9cb7bb4248e19552" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41723 } }
I20260621 02:55:09.395304 31072 leader_election.cc:302] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6921e7cccf764e9a9cb7bb4248e19552; no voters:
I20260621 02:55:09.395810 31072 leader_election.cc:288] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:09.395973 31077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:09.396605 31077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [term 1 LEADER]: Becoming Leader. State: Replica: 6921e7cccf764e9a9cb7bb4248e19552, State: Running, Role: LEADER
I20260621 02:55:09.397153 31077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6921e7cccf764e9a9cb7bb4248e19552" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41723 } }
I20260621 02:55:09.397917 31072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:09.399639 31079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6921e7cccf764e9a9cb7bb4248e19552. Latest consensus state: current_term: 1 leader_uuid: "6921e7cccf764e9a9cb7bb4248e19552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6921e7cccf764e9a9cb7bb4248e19552" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41723 } } }
I20260621 02:55:09.400002 31079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:09.400244 31078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6921e7cccf764e9a9cb7bb4248e19552" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6921e7cccf764e9a9cb7bb4248e19552" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41723 } } }
I20260621 02:55:09.400693 31078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:09.411851 31082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:09.418187 31082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:09.426023 31082 catalog_manager.cc:1347] Generated new cluster ID: 0938d8617b9d48a380a55e6c56ab571f
I20260621 02:55:09.426131 31082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:09.439612 31082 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:09.440944 31082 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:09.452332 31082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552: Generated new TSK 0
I20260621 02:55:09.453539 31082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
Entry for principal kudu/127.11.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab.
I20260621 02:55:09.511972 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:41723
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:09.834178 31099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:09.834438 31099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:09.834530 31099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:09.841503 31099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:09.841624 31099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:09.841814 31099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:55:09.851073 31099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.1.keytab
--principal=kudu/127.11.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:09.852310 31099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:09.853626 31099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:09.866248 31104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:09.866294 31107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:09.866375 31105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:09.867408 31099 server_base.cc:1034] running on GCE node
I20260621 02:55:09.867933 31099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:09.868711 31099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:09.869954 31099 hybrid_clock.cc:648] HybridClock initialized: now 1782010509869894 us; error 51 us; skew 500 ppm
I20260621 02:55:09.870247 31099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:09 dist-test-slave-bc52 krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010509, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:09.873400 31099 init.cc:377] Logged in from keytab as kudu/127.11.109.1@KRBTEST.COM (short username kudu)
I20260621 02:55:09.874784 31099 webserver.cc:460] Webserver started at http://127.11.109.1:34953/ using document root <none> and password file <none>
I20260621 02:55:09.875427 31099 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:09.875567 31099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:09.875887 31099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:09.878674 31099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "87e39b270a7e493b81418064410c6593"
format_stamp: "Formatted at 2026-06-21 02:55:09 on dist-test-slave-bc52"
server_key: "eaa9ae8f390da2fe7f809cfb82461640"
server_key_iv: "841aa0ad67bea0c93d28bf468e81884c"
server_key_version: "encryptionkey@0"
I20260621 02:55:09.879407 31099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "87e39b270a7e493b81418064410c6593"
format_stamp: "Formatted at 2026-06-21 02:55:09 on dist-test-slave-bc52"
server_key: "eaa9ae8f390da2fe7f809cfb82461640"
server_key_iv: "841aa0ad67bea0c93d28bf468e81884c"
server_key_version: "encryptionkey@0"
I20260621 02:55:09.884904 31099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260621 02:55:09.888262 31114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:09.889696 31099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:09.889859 31099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "87e39b270a7e493b81418064410c6593"
format_stamp: "Formatted at 2026-06-21 02:55:09 on dist-test-slave-bc52"
server_key: "eaa9ae8f390da2fe7f809cfb82461640"
server_key_iv: "841aa0ad67bea0c93d28bf468e81884c"
server_key_version: "encryptionkey@0"
I20260621 02:55:09.890067 31099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:09.918855 31099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:09.922874 31099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:09.923156 31099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:09.924081 31099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:09.925426 31099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:09.925534 31099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:09.925674 31099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:09.925747 31099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:09.956779 31099 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:36001
I20260621 02:55:09.956871 31226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:36001 every 8 connection(s)
I20260621 02:55:09.958060 31099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:55:09.964293 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31099
I20260621 02:55:09.964484 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:55:09.964900 11700 external_mini_cluster.cc:1428] Setting key c08384a5132788d455aab6d1a86c3c6a
Jun 21 02:55:09 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010509, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.1@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:09.978729 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:09.960596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:42499 (local address 127.11.109.62:41723)
0621 02:55:09.961066 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:09.961070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:09.961925 (+ 855us) server_negotiation.cc:407] Connection header received
0621 02:55:09.963739 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:09.963745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:09.963834 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:09.963961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:09.965620 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:09.966400 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:09.967930 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:09.968317 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:09.968458 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:09.972344 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:09.972348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:09.972351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:09.972383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:09.975471 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:09.976168 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:09.976176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:09.976183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:09.976329 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:09.976713 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:09.976721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:09.976727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:09.977102 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:09.977276 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:09.977977 (+ 701us) server_negotiation.cc:299] Negotiation successful
0621 02:55:09.978235 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260621 02:55:09.979281 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:09.961035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41723 (local address 127.11.109.1:42499)
0621 02:55:09.961770 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:09.961811 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:09.963481 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:09.964190 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:09.964198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:09.964791 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:09.965454 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:09.965472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:09.966554 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:09.966558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:09.967771 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:09.967786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:09.968552 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:09.968560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:09.968782 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:09.969495 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:09.969516 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:09.972143 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:09.975605 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:09.975612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:09.975625 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:09.976040 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:09.976431 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:09.976434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:09.976436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:09.976603 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:09.977281 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:09.977292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:09.977585 (+ 293us) client_negotiation.cc:769] Sending connection context
0621 02:55:09.977801 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 02:55:09.978182 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":322,"threads_started":1}
I20260621 02:55:09.980849 31227 heartbeater.cc:347] Connected to a master server at 127.11.109.62:41723
I20260621 02:55:09.981199 31227 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:09.982048 31227 heartbeater.cc:511] Master 127.11.109.62:41723 requested a full tablet report, sending...
I20260621 02:55:09.984234 31037 ts_manager.cc:194] Registered new tserver with Master: 87e39b270a7e493b81418064410c6593 (127.11.109.1:36001)
I20260621 02:55:09.985757 31037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.1@KRBTEST.COM'} at 127.11.109.1:42499
WARNING: no policy specified for kudu/127.11.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
Entry for principal kudu/127.11.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab.
I20260621 02:55:10.019928 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:41723
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:10.336679 31233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:10.336974 31233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:10.337137 31233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:10.344476 31233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:10.344627 31233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:10.344892 31233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:55:10.354851 31233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.2.keytab
--principal=kudu/127.11.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:10.356135 31233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:10.357563 31233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:10.372390 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:10.372344 31239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:10.372825 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:10.373588 31233 server_base.cc:1034] running on GCE node
I20260621 02:55:10.374249 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:10.375319 31233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:10.376550 31233 hybrid_clock.cc:648] HybridClock initialized: now 1782010510376485 us; error 58 us; skew 500 ppm
I20260621 02:55:10.376942 31233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:10 dist-test-slave-bc52 krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:10.380162 31233 init.cc:377] Logged in from keytab as kudu/127.11.109.2@KRBTEST.COM (short username kudu)
I20260621 02:55:10.381628 31233 webserver.cc:460] Webserver started at http://127.11.109.2:41395/ using document root <none> and password file <none>
I20260621 02:55:10.382248 31233 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:10.382352 31233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:10.382594 31233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:10.385377 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "6866c4097ff34c24b20e27421f5a1efc"
format_stamp: "Formatted at 2026-06-21 02:55:10 on dist-test-slave-bc52"
server_key: "59ed5db32c4791f52f1026623420aec7"
server_key_iv: "f838487ec9f182d5841a6b1571086ac3"
server_key_version: "encryptionkey@0"
I20260621 02:55:10.386073 31233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "6866c4097ff34c24b20e27421f5a1efc"
format_stamp: "Formatted at 2026-06-21 02:55:10 on dist-test-slave-bc52"
server_key: "59ed5db32c4791f52f1026623420aec7"
server_key_iv: "f838487ec9f182d5841a6b1571086ac3"
server_key_version: "encryptionkey@0"
I20260621 02:55:10.390810 31233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:55:10.394167 31248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:10.395467 31233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:10.395588 31233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "6866c4097ff34c24b20e27421f5a1efc"
format_stamp: "Formatted at 2026-06-21 02:55:10 on dist-test-slave-bc52"
server_key: "59ed5db32c4791f52f1026623420aec7"
server_key_iv: "f838487ec9f182d5841a6b1571086ac3"
server_key_version: "encryptionkey@0"
I20260621 02:55:10.395736 31233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:10.426757 31233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:10.430837 31233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:10.431084 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:10.432054 31233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:10.433425 31233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:10.433499 31233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:10.433588 31233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:10.433631 31233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:10.466212 31233 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:35435
I20260621 02:55:10.466249 31360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:35435 every 8 connection(s)
I20260621 02:55:10.467547 31233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:55:10.472594 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31233
I20260621 02:55:10.472782 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:55:10.473264 11700 external_mini_cluster.cc:1428] Setting key 73c77799066dbbdf053a0c481e0a84ed
Jun 21 02:55:10 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.2@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:10.487733 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:10.470057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:56817 (local address 127.11.109.62:41723)
0621 02:55:10.470232 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:10.470236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:10.471335 (+ 1099us) server_negotiation.cc:407] Connection header received
0621 02:55:10.473299 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:10.473306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:10.473431 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:10.473547 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:10.475330 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.476099 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.477746 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.478178 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.478309 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:10.481971 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:10.481976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:10.481979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:10.482014 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:10.484655 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:10.485449 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:10.485467 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:10.485474 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:10.485596 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:10.486028 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:10.486037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:10.486039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:10.486292 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:10.486444 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:10.487153 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 02:55:10.487379 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:55:10.488701 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:10.470497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41723 (local address 127.11.109.2:56817)
0621 02:55:10.471167 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:10.471210 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:10.472980 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:10.473792 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:10.473801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:10.474438 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:10.475160 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:10.475181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.476260 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.476265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:10.477595 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:10.477608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.478419 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.478426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:10.478656 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:10.479352 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:10.479376 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:10.481724 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:10.484805 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:10.484814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:10.484831 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:10.485326 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:10.485706 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:10.485712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:10.485716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:10.485900 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:10.486421 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:10.486430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:10.486773 (+ 343us) client_negotiation.cc:769] Sending connection context
0621 02:55:10.487051 (+ 278us) client_negotiation.cc:240] Negotiation successful
0621 02:55:10.487580 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":280,"threads_started":1}
I20260621 02:55:10.490228 31361 heartbeater.cc:347] Connected to a master server at 127.11.109.62:41723
I20260621 02:55:10.490571 31361 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:10.491406 31361 heartbeater.cc:511] Master 127.11.109.62:41723 requested a full tablet report, sending...
I20260621 02:55:10.493019 31037 ts_manager.cc:194] Registered new tserver with Master: 6866c4097ff34c24b20e27421f5a1efc (127.11.109.2:35435)
I20260621 02:55:10.494311 31037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.2@KRBTEST.COM'} at 127.11.109.2:56817
WARNING: no policy specified for kudu/127.11.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
Entry for principal kudu/127.11.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab.
I20260621 02:55:10.527673 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:41723
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:10.848999 31366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:10.849298 31366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:10.849395 31366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:10.856069 31366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:10.856156 31366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:10.856261 31366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:55:10.865401 31366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.3.keytab
--principal=kudu/127.11.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:41723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:10.866513 31366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:10.867810 31366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:10.880343 31374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:10.880321 31372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:10.880563 31371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:10.881138 31366 server_base.cc:1034] running on GCE node
I20260621 02:55:10.881745 31366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:10.882619 31366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:10.883829 31366 hybrid_clock.cc:648] HybridClock initialized: now 1782010510883769 us; error 64 us; skew 500 ppm
I20260621 02:55:10.884114 31366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:10 dist-test-slave-bc52 krb5kdc[30992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:10.891767 31366 init.cc:377] Logged in from keytab as kudu/127.11.109.3@KRBTEST.COM (short username kudu)
I20260621 02:55:10.893342 31366 webserver.cc:460] Webserver started at http://127.11.109.3:46243/ using document root <none> and password file <none>
I20260621 02:55:10.894052 31366 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:10.894196 31366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:10.894562 31366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:10.897312 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "0d06852b2a4c45cc846bf29eba7c9cab"
format_stamp: "Formatted at 2026-06-21 02:55:10 on dist-test-slave-bc52"
server_key: "9551e44e4a294144fc1b4c06d0a28143"
server_key_iv: "aa904e0c74404e1b57782fbec64d2fb8"
server_key_version: "encryptionkey@0"
I20260621 02:55:10.898168 31366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "0d06852b2a4c45cc846bf29eba7c9cab"
format_stamp: "Formatted at 2026-06-21 02:55:10 on dist-test-slave-bc52"
server_key: "9551e44e4a294144fc1b4c06d0a28143"
server_key_iv: "aa904e0c74404e1b57782fbec64d2fb8"
server_key_version: "encryptionkey@0"
I20260621 02:55:10.903232 31366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:55:10.906498 31381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:10.907647 31366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:55:10.907810 31366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "0d06852b2a4c45cc846bf29eba7c9cab"
format_stamp: "Formatted at 2026-06-21 02:55:10 on dist-test-slave-bc52"
server_key: "9551e44e4a294144fc1b4c06d0a28143"
server_key_iv: "aa904e0c74404e1b57782fbec64d2fb8"
server_key_version: "encryptionkey@0"
I20260621 02:55:10.907996 31366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:10.928067 31366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:10.931785 31366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:10.932060 31366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:10.932978 31366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:10.934222 31366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:10.934320 31366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:10.934468 31366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:10.934540 31366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:10.967517 31366 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:38891
I20260621 02:55:10.967555 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:38891 every 8 connection(s)
I20260621 02:55:10.968696 31366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:55:10.970592 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31366
I20260621 02:55:10.970785 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:55:10.971287 11700 external_mini_cluster.cc:1428] Setting key bf7bce6460036b6ed631662cfa88ab69
Jun 21 02:55:10 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010510, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.3@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:10.987866 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:10.971367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:40195 (local address 127.11.109.62:41723)
0621 02:55:10.971584 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:10.971589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:10.972960 (+ 1371us) server_negotiation.cc:407] Connection header received
0621 02:55:10.974922 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:10.974926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:10.974989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:10.975076 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:10.976442 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.977259 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.978735 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.979138 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.979228 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:10.982512 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:10.982518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:10.982521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:10.982551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:10.985013 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:10.985863 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:10.985869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:10.985879 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:10.986017 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:10.986510 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:10.986515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:10.986519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:10.986779 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:10.986887 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:10.987476 (+ 589us) server_negotiation.cc:299] Negotiation successful
0621 02:55:10.987625 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:55:10.988716 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:10.971906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41723 (local address 127.11.109.3:40195)
0621 02:55:10.972792 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:10.972842 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:10.974676 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:10.975271 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:10.975279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:10.975766 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:10.976277 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:10.976293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.977389 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.977392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:10.978611 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:10.978622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:10.979310 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:10.979314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:10.979503 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:10.980061 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:10.980086 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:10.982285 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:10.985141 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:10.985155 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:10.985165 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:10.985764 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:10.986129 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:10.986133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:10.986136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:10.986366 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:10.986882 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:10.986888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:10.987161 (+ 273us) client_negotiation.cc:769] Sending connection context
0621 02:55:10.987374 (+ 213us) client_negotiation.cc:240] Negotiation successful
0621 02:55:10.987713 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":317,"threads_started":1}
I20260621 02:55:10.989486 31227 heartbeater.cc:503] Master 127.11.109.62:41723 was elected leader, sending a full tablet report...
I20260621 02:55:10.990119 31494 heartbeater.cc:347] Connected to a master server at 127.11.109.62:41723
I20260621 02:55:10.990393 31494 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:10.991036 31494 heartbeater.cc:511] Master 127.11.109.62:41723 requested a full tablet report, sending...
I20260621 02:55:10.992304 31037 ts_manager.cc:194] Registered new tserver with Master: 0d06852b2a4c45cc846bf29eba7c9cab (127.11.109.3:38891)
I20260621 02:55:10.993034 31037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.109.3@KRBTEST.COM'} at 127.11.109.3:40195
I20260621 02:55:10.998340 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:55:11.014695 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.002267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.11.109.62:41723)
0621 02:55:11.002413 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.002436 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.002847 (+ 411us) server_negotiation.cc:407] Connection header received
0621 02:55:11.003035 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.003044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.003096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.003190 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:11.004435 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.005146 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.006492 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.006953 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.007087 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.009360 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:11.009365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:11.009368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:11.009402 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:11.012086 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.012860 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.012872 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.012874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.013010 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.013430 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.013433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.013435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.013750 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:11.013935 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.014211 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.014363 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:55:11.018450 31037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:55:11.025146 31037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:55:11.047451 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.036939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.11.109.3:38891)
0621 02:55:11.039345 (+ 2406us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.039357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.039393 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:55:11.039505 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.039515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.039809 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.040732 (+ 923us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:11.041034 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.042544 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.045254 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.046320 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.046470 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.046699 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.046937 (+ 238us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.047093 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2203,"thread_start_us":1530,"threads_started":1}
I20260621 02:55:11.047537 31507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.036622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:38891 (local address 127.0.0.1:56486)
0621 02:55:11.037931 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:11.037976 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:11.038172 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:11.040060 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:11.040067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:11.040100 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:11.040523 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.040536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.042665 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.042673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.045035 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.045052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.046504 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.046510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.046692 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.046740 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 02:55:11.046899 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 02:55:11.047075 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":25,"thread_start_us":1607,"threads_started":1}
I20260621 02:55:11.048188 31509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.037329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:35435 (local address 127.0.0.1:49446)
0621 02:55:11.040136 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:11.040156 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:11.040276 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:11.040894 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:11.040901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:11.040929 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:11.041350 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.041364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.043266 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.043270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.045243 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.045264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.047321 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.047327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.047524 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.047562 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:55:11.047660 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:55:11.047780 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2687,"mutex_wait_us":131,"thread_start_us":2560,"threads_started":1}
I20260621 02:55:11.048507 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.036938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:36001 (local address 127.0.0.1:55260)
0621 02:55:11.038519 (+ 1581us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:11.038538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:11.038659 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:11.040528 (+ 1869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:11.040533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:11.040564 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:11.041054 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.041069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.043208 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.043213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.045631 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.045645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.047666 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.047672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.047818 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.047847 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 02:55:11.047928 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 02:55:11.048055 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":64,"thread_start_us":527,"threads_started":1}
I20260621 02:55:11.048755 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.037443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.11.109.1:36001)
0621 02:55:11.039875 (+ 2432us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.039884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.039915 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:55:11.040031 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.040038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.040269 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.040557 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:11.041491 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.043067 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.045771 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.047171 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.047310 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.047520 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.048301 (+ 781us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.048458 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2272,"thread_start_us":2112,"threads_started":1}
I20260621 02:55:11.048844 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.037763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.11.109.2:35435)
0621 02:55:11.039377 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.039388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.040261 (+ 873us) server_negotiation.cc:407] Connection header received
0621 02:55:11.040452 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.040460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.040698 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.040875 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:11.041515 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.043078 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.045432 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.046866 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.047004 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.047173 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.048405 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.048581 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1418,"thread_start_us":1139,"threads_started":1}
I20260621 02:55:11.051862 31162 tablet_service.cc:1467] Processing CreateTablet for tablet 38456fbbff8147f3964470365379f52f (DEFAULT_TABLE table=test-table [id=6a4a3a3f5c054814b15c9751cdeb0070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:11.051724 31429 tablet_service.cc:1467] Processing CreateTablet for tablet 38456fbbff8147f3964470365379f52f (DEFAULT_TABLE table=test-table [id=6a4a3a3f5c054814b15c9751cdeb0070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:11.053012 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38456fbbff8147f3964470365379f52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:11.053122 31429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38456fbbff8147f3964470365379f52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:11.053062 31296 tablet_service.cc:1467] Processing CreateTablet for tablet 38456fbbff8147f3964470365379f52f (DEFAULT_TABLE table=test-table [id=6a4a3a3f5c054814b15c9751cdeb0070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:11.054275 31296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38456fbbff8147f3964470365379f52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:11.063854 31512 tablet_bootstrap.cc:492] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Bootstrap starting.
I20260621 02:55:11.066826 31513 tablet_bootstrap.cc:492] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Bootstrap starting.
I20260621 02:55:11.067936 31514 tablet_bootstrap.cc:492] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Bootstrap starting.
I20260621 02:55:11.068869 31512 tablet_bootstrap.cc:654] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:11.070217 31513 tablet_bootstrap.cc:654] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:11.070179 31512 log.cc:824] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:11.070650 31514 tablet_bootstrap.cc:654] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:11.071342 31513 log.cc:824] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:11.071872 31514 log.cc:824] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:11.073159 31512 tablet_bootstrap.cc:492] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: No bootstrap required, opened a new log
I20260621 02:55:11.073441 31512 ts_tablet_manager.cc:1397] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260621 02:55:11.074239 31513 tablet_bootstrap.cc:492] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: No bootstrap required, opened a new log
I20260621 02:55:11.074512 31513 ts_tablet_manager.cc:1397] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:55:11.082888 31514 tablet_bootstrap.cc:492] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: No bootstrap required, opened a new log
I20260621 02:55:11.083125 31514 ts_tablet_manager.cc:1397] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20260621 02:55:11.083745 31512 raft_consensus.cc:348] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.084149 31512 raft_consensus.cc:374] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:11.084250 31512 raft_consensus.cc:729] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e39b270a7e493b81418064410c6593, State: Initialized, Role: FOLLOWER
I20260621 02:55:11.084671 31513 raft_consensus.cc:348] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.084882 31512 consensus_queue.cc:260] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.085112 31513 raft_consensus.cc:374] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:11.085249 31513 raft_consensus.cc:729] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d06852b2a4c45cc846bf29eba7c9cab, State: Initialized, Role: FOLLOWER
I20260621 02:55:11.085852 31513 consensus_queue.cc:260] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.086627 31512 ts_tablet_manager.cc:1428] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260621 02:55:11.087744 31494 heartbeater.cc:503] Master 127.11.109.62:41723 was elected leader, sending a full tablet report...
I20260621 02:55:11.088327 31513 ts_tablet_manager.cc:1428] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260621 02:55:11.092674 31514 raft_consensus.cc:348] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.092950 31514 raft_consensus.cc:374] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:11.093058 31514 raft_consensus.cc:729] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6866c4097ff34c24b20e27421f5a1efc, State: Initialized, Role: FOLLOWER
I20260621 02:55:11.093534 31514 consensus_queue.cc:260] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.094421 31361 heartbeater.cc:503] Master 127.11.109.62:41723 was elected leader, sending a full tablet report...
I20260621 02:55:11.094954 31514 ts_tablet_manager.cc:1428] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260621 02:55:11.210485 31228 tablet.cc:2369] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:55:11.219519 31362 tablet.cc:2369] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:55:11.220315 31495 tablet.cc:2369] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:55:11.335297 31518 raft_consensus.cc:482] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:55:11.335556 31518 raft_consensus.cc:504] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.337110 31518 leader_election.cc:288] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d06852b2a4c45cc846bf29eba7c9cab (127.11.109.3:38891), 6866c4097ff34c24b20e27421f5a1efc (127.11.109.2:35435)
I20260621 02:55:11.343497 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.337528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:38299 (local address 127.11.109.2:35435)
0621 02:55:11.337732 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.337739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.338301 (+ 562us) server_negotiation.cc:407] Connection header received
0621 02:55:11.338525 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.338532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.338628 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.338715 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:11.339279 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.340011 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.341854 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.342637 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.342828 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.342904 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.343130 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.343248 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:55:11.343541 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.337515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:53009 (local address 127.11.109.3:38891)
0621 02:55:11.337712 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.337718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.338514 (+ 796us) server_negotiation.cc:407] Connection header received
0621 02:55:11.338700 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.338705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.338777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.338872 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:11.339491 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.340184 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.341890 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.342639 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.342804 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.342873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.343189 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.343305 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:55:11.343653 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.337645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:38891 (local address 127.11.109.1:53009)
0621 02:55:11.338399 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:11.338412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:11.338548 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:11.338897 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:11.338900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:11.338949 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:11.339319 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.339333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.340313 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.340316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.341727 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.341738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.342893 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.342898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.343031 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.343050 (+ 19us) client_negotiation.cc:769] Sending connection context
0621 02:55:11.343151 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 02:55:11.343254 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":3,"spinlock_wait_cycles":5504,"thread_start_us":417,"threads_started":1}
I20260621 02:55:11.343804 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.337359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:35435 (local address 127.11.109.1:38299)
0621 02:55:11.338176 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:11.338200 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:11.338363 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:11.338743 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:11.338748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:11.338772 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:11.339140 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.339150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.340127 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.340135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.341715 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:11.341726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.342842 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.342847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:11.342995 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.343016 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:55:11.343085 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 02:55:11.343201 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":54,"thread_start_us":419,"threads_started":1}
I20260621 02:55:11.344643 31316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38456fbbff8147f3964470365379f52f" candidate_uuid: "87e39b270a7e493b81418064410c6593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6866c4097ff34c24b20e27421f5a1efc" is_pre_election: true
I20260621 02:55:11.344645 31449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38456fbbff8147f3964470365379f52f" candidate_uuid: "87e39b270a7e493b81418064410c6593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" is_pre_election: true
I20260621 02:55:11.345213 31449 raft_consensus.cc:2454] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e39b270a7e493b81418064410c6593 in term 0.
I20260621 02:55:11.345216 31316 raft_consensus.cc:2454] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e39b270a7e493b81418064410c6593 in term 0.
I20260621 02:55:11.345824 31118 leader_election.cc:302] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6866c4097ff34c24b20e27421f5a1efc, 87e39b270a7e493b81418064410c6593; no voters:
I20260621 02:55:11.346217 31518 raft_consensus.cc:2781] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:55:11.346370 31518 raft_consensus.cc:482] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:55:11.346487 31518 raft_consensus.cc:3037] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:11.348577 31518 raft_consensus.cc:504] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.349226 31518 leader_election.cc:288] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [CANDIDATE]: Term 1 election: Requested vote from peers 0d06852b2a4c45cc846bf29eba7c9cab (127.11.109.3:38891), 6866c4097ff34c24b20e27421f5a1efc (127.11.109.2:35435)
I20260621 02:55:11.349519 31449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38456fbbff8147f3964470365379f52f" candidate_uuid: "87e39b270a7e493b81418064410c6593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d06852b2a4c45cc846bf29eba7c9cab"
I20260621 02:55:11.349697 31449 raft_consensus.cc:3037] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:11.349643 31316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38456fbbff8147f3964470365379f52f" candidate_uuid: "87e39b270a7e493b81418064410c6593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6866c4097ff34c24b20e27421f5a1efc"
I20260621 02:55:11.349848 31316 raft_consensus.cc:3037] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:11.351785 31449 raft_consensus.cc:2454] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e39b270a7e493b81418064410c6593 in term 1.
I20260621 02:55:11.351966 31316 raft_consensus.cc:2454] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e39b270a7e493b81418064410c6593 in term 1.
I20260621 02:55:11.352162 31116 leader_election.cc:302] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d06852b2a4c45cc846bf29eba7c9cab, 87e39b270a7e493b81418064410c6593; no voters:
I20260621 02:55:11.352442 31518 raft_consensus.cc:2781] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:11.352869 31518 raft_consensus.cc:686] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 1 LEADER]: Becoming Leader. State: Replica: 87e39b270a7e493b81418064410c6593, State: Running, Role: LEADER
I20260621 02:55:11.353379 31518 consensus_queue.cc:237] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } }
I20260621 02:55:11.357657 31037 catalog_manager.cc:5495] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e39b270a7e493b81418064410c6593 (127.11.109.1). New cstate: current_term: 1 leader_uuid: "87e39b270a7e493b81418064410c6593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87e39b270a7e493b81418064410c6593" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 } health_report { overall_health: UNKNOWN } } }
I20260621 02:55:11.410203 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.404977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.11.109.1:36001)
0621 02:55:11.405208 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.405216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.405241 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:55:11.405495 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.405499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.405580 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.405728 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:11.406104 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.406813 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.408133 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.408548 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.408693 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.409159 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:11.409653 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:11.409817 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.409866 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.409943 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:11.418205 31449 raft_consensus.cc:1261] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 1 FOLLOWER]: Refusing update from remote peer 87e39b270a7e493b81418064410c6593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:11.418267 31316 raft_consensus.cc:1261] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 1 FOLLOWER]: Refusing update from remote peer 87e39b270a7e493b81418064410c6593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:11.419203 31518 consensus_queue.cc:1035] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6866c4097ff34c24b20e27421f5a1efc" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:11.419523 31523 consensus_queue.cc:1035] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d06852b2a4c45cc846bf29eba7c9cab" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:11.432458 31527 mvcc.cc:205] Tried to move back new op lower bound from 7299115054758797312 to 7299115054507130880. Current Snapshot: MvccSnapshot[applied={T|T < 7299115054758797312}]
I20260621 02:55:11.433861 31526 mvcc.cc:205] Tried to move back new op lower bound from 7299115054758797312 to 7299115054507130880. Current Snapshot: MvccSnapshot[applied={T|T < 7299115054758797312 or (T in {7299115054758797312})}]
I20260621 02:55:11.442169 31037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:11.442466 31037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:11.447034 31037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6921e7cccf764e9a9cb7bb4248e19552: Sending DeleteTablet for 3 replicas of tablet 38456fbbff8147f3964470365379f52f
I20260621 02:55:11.448242 31429 tablet_service.cc:1514] Processing DeleteTablet for tablet 38456fbbff8147f3964470365379f52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:11 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:56486
I20260621 02:55:11.448452 31162 tablet_service.cc:1514] Processing DeleteTablet for tablet 38456fbbff8147f3964470365379f52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:11 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:55260
I20260621 02:55:11.448599 31296 tablet_service.cc:1514] Processing DeleteTablet for tablet 38456fbbff8147f3964470365379f52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:11 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:49446
I20260621 02:55:11.449147 31295 tablet_service.cc:1514] Processing DeleteTablet for tablet 38456fbbff8147f3964470365379f52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:11 UTC) from {username='kudu', principal='kudu/127.11.109.62@KRBTEST.COM'} at 127.0.0.1:49446
I20260621 02:55:11.450670 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31099
I20260621 02:55:11.458530 31536 tablet_replica.cc:330] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: stopping tablet replica
I20260621 02:55:11.459052 31536 raft_consensus.cc:2229] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:11.459611 31536 raft_consensus.cc:2258] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:11.459810 31537 tablet_replica.cc:330] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: stopping tablet replica
I20260621 02:55:11.460904 31537 raft_consensus.cc:2229] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:55:11.461967 31536 ts_tablet_manager.cc:1905] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:11.462608 31537 raft_consensus.cc:2258] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:11.462756 31538 tablet_replica.cc:330] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: stopping tablet replica
I20260621 02:55:11.463109 31538 raft_consensus.cc:2229] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:11.463584 31538 raft_consensus.cc:2258] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:11.466897 31538 ts_tablet_manager.cc:1905] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:11.467804 31537 ts_tablet_manager.cc:1905] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:11.469341 31536 ts_tablet_manager.cc:1918] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:11.469520 31536 log.cc:1196] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/38456fbbff8147f3964470365379f52f
Jun 21 02:55:11 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.1@KRBTEST.COM
I20260621 02:55:11.470026 31536 ts_tablet_manager.cc:1939] T 38456fbbff8147f3964470365379f52f P 0d06852b2a4c45cc846bf29eba7c9cab: Deleting consensus metadata
I20260621 02:55:11.471916 31022 catalog_manager.cc:4841] TS 0d06852b2a4c45cc846bf29eba7c9cab (127.11.109.3:38891): tablet 38456fbbff8147f3964470365379f52f (table test-table [id=6a4a3a3f5c054814b15c9751cdeb0070]) successfully deleted
I20260621 02:55:11.478083 31538 ts_tablet_manager.cc:1918] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:11.478250 31538 log.cc:1196] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/38456fbbff8147f3964470365379f52f
I20260621 02:55:11.478816 31538 ts_tablet_manager.cc:1939] T 38456fbbff8147f3964470365379f52f P 6866c4097ff34c24b20e27421f5a1efc: Deleting consensus metadata
I20260621 02:55:11.478799 31537 ts_tablet_manager.cc:1918] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:11.479043 31537 log.cc:1196] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/38456fbbff8147f3964470365379f52f
I20260621 02:55:11.479637 31537 ts_tablet_manager.cc:1939] T 38456fbbff8147f3964470365379f52f P 87e39b270a7e493b81418064410c6593: Deleting consensus metadata
I20260621 02:55:11.479733 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.452614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.11.109.1:36001)
0621 02:55:11.452820 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.452828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.457765 (+ 4937us) server_negotiation.cc:407] Connection header received
0621 02:55:11.457983 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.457991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.458091 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.458198 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:11.461242 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.462653 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.464621 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.465361 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.465461 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.470907 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:11.470920 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:11.470947 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:11.471013 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:11.476078 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.477004 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.477012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.477016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.477202 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.477651 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.477655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.477658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.478128 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:11.478454 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.479119 (+ 665us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.479402 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:55:11.480791 31024 catalog_manager.cc:4841] TS 6866c4097ff34c24b20e27421f5a1efc (127.11.109.2:35435): tablet 38456fbbff8147f3964470365379f52f (table test-table [id=6a4a3a3f5c054814b15c9751cdeb0070]) successfully deleted
I20260621 02:55:11.481065 31222 generic_service.cc:196] Checking for leaks (request via RPC)
W20260621 02:55:11.481823 31024 catalog_manager.cc:4808] TS 6866c4097ff34c24b20e27421f5a1efc (127.11.109.2:35435): delete failed for tablet 38456fbbff8147f3964470365379f52f because the tablet was not found. No further retry: Not found: Tablet not found: 38456fbbff8147f3964470365379f52f
I20260621 02:55:11.578859 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31099
I20260621 02:55:11.579119 31023 catalog_manager.cc:4841] TS 87e39b270a7e493b81418064410c6593 (127.11.109.1:36001): tablet 38456fbbff8147f3964470365379f52f (table test-table [id=6a4a3a3f5c054814b15c9751cdeb0070]) successfully deleted
I20260621 02:55:11.596621 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31233
Jun 21 02:55:11 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.2@KRBTEST.COM
I20260621 02:55:11.611194 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.597367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.11.109.2:35435)
0621 02:55:11.597556 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.597563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.597657 (+ 94us) server_negotiation.cc:407] Connection header received
0621 02:55:11.597867 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.597871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.597978 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.598089 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:11.599462 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.600275 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.601713 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.602127 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.602287 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.606438 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:11.606450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:11.606464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:11.606495 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:11.608971 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.609628 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.609635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.609637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.609759 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.610054 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.610057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.610058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.610357 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:11.610552 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.610748 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.610901 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:55:11.611862 31356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:11.699532 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31233
I20260621 02:55:11.716785 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31366
Jun 21 02:55:11 dist-test-slave-bc52 krb5kdc[30992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.3@KRBTEST.COM
I20260621 02:55:11.731747 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:11.717595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.11.109.3:38891)
0621 02:55:11.717771 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:11.717777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:11.717818 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:55:11.718034 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:11.718039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:11.718128 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:11.718237 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:11.719501 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.720317 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.721765 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:11.722164 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:11.722292 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:11.726529 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:11.726549 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:11.726566 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:11.726596 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:11.729030 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.729720 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.729729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.729732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.729880 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:11.730254 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:11.730260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:11.730263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:11.730601 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:11.730879 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:11.731188 (+ 309us) server_negotiation.cc:299] Negotiation successful
0621 02:55:11.731432 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:55:11.732359 31489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:11.823550 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31366
I20260621 02:55:11.839900 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31005
I20260621 02:55:11.841032 31067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:11.949594 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31005
2026-06-21T02:55:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4355 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T02:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:12Z Disabled control of system clock
I20260621 02:55:12.025164 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44675
--webserver_interface=127.11.109.62
--webserver_port=0
--builtin_ntp_servers=127.11.109.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:55:12.343915 31547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:12.344178 31547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:12.344333 31547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:12.351222 31547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:12.351311 31547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:12.351395 31547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:12.351490 31547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:12.351548 31547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:12.360102 31547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:44675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:12.361313 31547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:12.363047 31547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:12.374500 31553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:12.374469 31555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:12.375011 31552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:12.375360 31547 server_base.cc:1034] running on GCE node
I20260621 02:55:12.376055 31547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:12.377293 31547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:12.378490 31547 hybrid_clock.cc:648] HybridClock initialized: now 1782010512378448 us; error 42 us; skew 500 ppm
I20260621 02:55:12.378783 31547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:12.380750 31547 webserver.cc:460] Webserver started at http://127.11.109.62:33789/ using document root <none> and password file <none>
I20260621 02:55:12.381385 31547 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:12.381493 31547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:12.381748 31547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:12.384421 31547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "74e4998b1a914589ac62348237061adc"
format_stamp: "Formatted at 2026-06-21 02:55:12 on dist-test-slave-bc52"
server_key: "263b6059f3c9c8da2fcf57a91ef58605"
server_key_iv: "e04ba9b3616b069d5b44e3aed1f2a989"
server_key_version: "encryptionkey@0"
I20260621 02:55:12.385150 31547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "74e4998b1a914589ac62348237061adc"
format_stamp: "Formatted at 2026-06-21 02:55:12 on dist-test-slave-bc52"
server_key: "263b6059f3c9c8da2fcf57a91ef58605"
server_key_iv: "e04ba9b3616b069d5b44e3aed1f2a989"
server_key_version: "encryptionkey@0"
I20260621 02:55:12.389807 31547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:55:12.392880 31561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:12.394126 31547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260621 02:55:12.394271 31547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "74e4998b1a914589ac62348237061adc"
format_stamp: "Formatted at 2026-06-21 02:55:12 on dist-test-slave-bc52"
server_key: "263b6059f3c9c8da2fcf57a91ef58605"
server_key_iv: "e04ba9b3616b069d5b44e3aed1f2a989"
server_key_version: "encryptionkey@0"
I20260621 02:55:12.394435 31547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:12.411616 31547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:12.412313 31547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:12.412531 31547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:12.430297 31547 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:44675
I20260621 02:55:12.430320 31612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:44675 every 8 connection(s)
I20260621 02:55:12.431631 31547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:12.435156 31613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:12.435613 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31547
I20260621 02:55:12.435781 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:12.436259 11700 external_mini_cluster.cc:1428] Setting key 0c114a73d9e3e2f005e57d8334dfac2f
I20260621 02:55:12.449657 31613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc: Bootstrap starting.
I20260621 02:55:12.451639 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:12.438048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.11.109.62:44675)
0621 02:55:12.439076 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:12.439089 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:12.439138 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:55:12.441835 (+ 2697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:12.441863 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:12.442453 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:12.442881 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:12.443395 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:12.445311 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:12.446611 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:12.447667 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:12.447833 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:12.448884 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:12.448894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:12.448911 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:12.449615 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:12.449715 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:12.449742 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:12.449927 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:12.450104 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 02:55:12.450400 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":306,"threads_started":1}
I20260621 02:55:12.453531 31613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:12.454690 31613 log.cc:824] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:12.457298 31613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc: No bootstrap required, opened a new log
I20260621 02:55:12.464345 31613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e4998b1a914589ac62348237061adc" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44675 } }
I20260621 02:55:12.464704 31613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:12.464838 31613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e4998b1a914589ac62348237061adc, State: Initialized, Role: FOLLOWER
I20260621 02:55:12.465499 31613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e4998b1a914589ac62348237061adc" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44675 } }
I20260621 02:55:12.465775 31613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:12.465909 31613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:12.466078 31613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:12.468156 31613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e4998b1a914589ac62348237061adc" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44675 } }
I20260621 02:55:12.468636 31613 leader_election.cc:302] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74e4998b1a914589ac62348237061adc; no voters:
I20260621 02:55:12.469306 31613 leader_election.cc:288] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:12.469445 31618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:12.470139 31618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [term 1 LEADER]: Becoming Leader. State: Replica: 74e4998b1a914589ac62348237061adc, State: Running, Role: LEADER
I20260621 02:55:12.470669 31618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e4998b1a914589ac62348237061adc" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44675 } }
I20260621 02:55:12.471503 31613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:12.473574 31619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74e4998b1a914589ac62348237061adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e4998b1a914589ac62348237061adc" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44675 } } }
I20260621 02:55:12.473548 31620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74e4998b1a914589ac62348237061adc. Latest consensus state: current_term: 1 leader_uuid: "74e4998b1a914589ac62348237061adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e4998b1a914589ac62348237061adc" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 44675 } } }
I20260621 02:55:12.473989 31619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:12.474028 31620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:12.478407 31624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:12.483747 31624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:12.491772 31624 catalog_manager.cc:1347] Generated new cluster ID: afb9a7347f1441ba82030b09d968785a
I20260621 02:55:12.491874 31624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:12.525941 31624 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:12.527269 31624 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:12.537863 31624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc: Generated new TSK 0
I20260621 02:55:12.538705 31624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:12.552371 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.1:0
--local_ip_for_outbound_sockets=127.11.109.1
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44675
--builtin_ntp_servers=127.11.109.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:55:12.872038 31637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:12.872342 31637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:12.872495 31637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:12.879660 31637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:12.879798 31637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:12.879958 31637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.1
I20260621 02:55:12.888799 31637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.109.1
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.1
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:12.890156 31637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:12.891906 31637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:12.904735 31642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:12.904641 31643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:12.904708 31645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:12.905570 31637 server_base.cc:1034] running on GCE node
I20260621 02:55:12.906208 31637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:12.907065 31637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:12.908299 31637 hybrid_clock.cc:648] HybridClock initialized: now 1782010512908245 us; error 54 us; skew 500 ppm
I20260621 02:55:12.908607 31637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:12.910668 31637 webserver.cc:460] Webserver started at http://127.11.109.1:34459/ using document root <none> and password file <none>
I20260621 02:55:12.911259 31637 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:12.911340 31637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:12.911619 31637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:12.914330 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data/instance:
uuid: "3171452ca8db46f1a725856535340da5"
format_stamp: "Formatted at 2026-06-21 02:55:12 on dist-test-slave-bc52"
server_key: "7b26e8c8121ddd2f9206b3aabea67d4f"
server_key_iv: "071a4d1f969eb6930aa2e47b6c850514"
server_key_version: "encryptionkey@0"
I20260621 02:55:12.915072 31637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance:
uuid: "3171452ca8db46f1a725856535340da5"
format_stamp: "Formatted at 2026-06-21 02:55:12 on dist-test-slave-bc52"
server_key: "7b26e8c8121ddd2f9206b3aabea67d4f"
server_key_iv: "071a4d1f969eb6930aa2e47b6c850514"
server_key_version: "encryptionkey@0"
I20260621 02:55:12.919862 31637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:55:12.923205 31651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:12.924543 31637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:55:12.924700 31637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal
uuid: "3171452ca8db46f1a725856535340da5"
format_stamp: "Formatted at 2026-06-21 02:55:12 on dist-test-slave-bc52"
server_key: "7b26e8c8121ddd2f9206b3aabea67d4f"
server_key_iv: "071a4d1f969eb6930aa2e47b6c850514"
server_key_version: "encryptionkey@0"
I20260621 02:55:12.924912 31637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:12.959007 31637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:12.959848 31637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:12.960153 31637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:12.961104 31637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:12.962575 31637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:12.962700 31637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:12.962831 31637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:12.962917 31637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:12.995012 31637 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.1:45725
I20260621 02:55:12.995098 31763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.1:45725 every 8 connection(s)
I20260621 02:55:12.996266 31637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/data/info.pb
I20260621 02:55:12.998773 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31637
I20260621 02:55:12.998953 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal/instance
I20260621 02:55:12.999466 11700 external_mini_cluster.cc:1428] Setting key 510cc2e23837f705b82c9980948c5765
I20260621 02:55:13.002509 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.2:0
--local_ip_for_outbound_sockets=127.11.109.2
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44675
--builtin_ntp_servers=127.11.109.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:55:13.011174 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:12.998879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:54345 (local address 127.11.109.62:44675)
0621 02:55:12.999366 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:12.999372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.000769 (+ 1397us) server_negotiation.cc:407] Connection header received
0621 02:55:13.003223 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.003228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.003286 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.003372 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:13.004735 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.006000 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.008447 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.008820 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.008924 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.010499 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:13.010504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:13.010507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:13.010524 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:13.010561 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:13.010572 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:13.010639 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.010878 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.010948 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
I20260621 02:55:13.012415 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:12.999614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44675 (local address 127.11.109.1:54345)
0621 02:55:13.000631 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:13.000668 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:13.002944 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:13.003624 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:13.003638 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:13.003754 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:13.004530 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.004556 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.006162 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.006169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.008300 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.008317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.009045 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.009055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.009409 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.010156 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:55:13.010162 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:13.010215 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:55:13.010315 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:13.010665 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:13.010678 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:13.010688 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:55:13.010864 (+ 176us) client_negotiation.cc:240] Negotiation successful
0621 02:55:13.011271 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":312,"threads_started":1}
I20260621 02:55:13.014014 31764 heartbeater.cc:347] Connected to a master server at 127.11.109.62:44675
I20260621 02:55:13.014299 31764 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:13.014987 31764 heartbeater.cc:511] Master 127.11.109.62:44675 requested a full tablet report, sending...
I20260621 02:55:13.016937 31578 ts_manager.cc:194] Registered new tserver with Master: 3171452ca8db46f1a725856535340da5 (127.11.109.1:45725)
I20260621 02:55:13.018428 31578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.1:54345
W20260621 02:55:13.319036 31768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:13.319334 31768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:13.319509 31768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:13.327813 31768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:13.327925 31768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:13.328111 31768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.2
I20260621 02:55:13.337306 31768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.109.2
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.2
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:13.338516 31768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:13.340322 31768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:13.353348 31773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:13.353319 31776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:13.353421 31774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:13.354238 31768 server_base.cc:1034] running on GCE node
I20260621 02:55:13.354826 31768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:13.355702 31768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:13.356941 31768 hybrid_clock.cc:648] HybridClock initialized: now 1782010513356911 us; error 61 us; skew 500 ppm
I20260621 02:55:13.357309 31768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:13.359484 31768 webserver.cc:460] Webserver started at http://127.11.109.2:39411/ using document root <none> and password file <none>
I20260621 02:55:13.360153 31768 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:13.360277 31768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:13.360592 31768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:13.363399 31768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data/instance:
uuid: "f80ad02d33f14fd993b88499b79e6025"
format_stamp: "Formatted at 2026-06-21 02:55:13 on dist-test-slave-bc52"
server_key: "cac61163209a530a2213486f3be746c2"
server_key_iv: "81b8fd5d096d5366fa54f50c72d5f39e"
server_key_version: "encryptionkey@0"
I20260621 02:55:13.364171 31768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance:
uuid: "f80ad02d33f14fd993b88499b79e6025"
format_stamp: "Formatted at 2026-06-21 02:55:13 on dist-test-slave-bc52"
server_key: "cac61163209a530a2213486f3be746c2"
server_key_iv: "81b8fd5d096d5366fa54f50c72d5f39e"
server_key_version: "encryptionkey@0"
I20260621 02:55:13.369040 31768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:55:13.372341 31782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:13.373646 31768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:13.373816 31768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal
uuid: "f80ad02d33f14fd993b88499b79e6025"
format_stamp: "Formatted at 2026-06-21 02:55:13 on dist-test-slave-bc52"
server_key: "cac61163209a530a2213486f3be746c2"
server_key_iv: "81b8fd5d096d5366fa54f50c72d5f39e"
server_key_version: "encryptionkey@0"
I20260621 02:55:13.374025 31768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:13.399726 31768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:13.400496 31768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:13.400782 31768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:13.401862 31768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:13.403204 31768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:13.403332 31768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:13.403469 31768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:13.403558 31768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:13.436756 31768 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.2:46749
I20260621 02:55:13.436839 31894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.2:46749 every 8 connection(s)
I20260621 02:55:13.438125 31768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/data/info.pb
I20260621 02:55:13.438592 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31768
I20260621 02:55:13.438747 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal/instance
I20260621 02:55:13.439293 11700 external_mini_cluster.cc:1428] Setting key e0ec3b490ab079200839624511cd6ce8
I20260621 02:55:13.442333 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.109.3:0
--local_ip_for_outbound_sockets=127.11.109.3
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44675
--builtin_ntp_servers=127.11.109.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:55:13.453610 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.441138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.2:46661 (local address 127.11.109.62:44675)
0621 02:55:13.441334 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:13.441340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.443089 (+ 1749us) server_negotiation.cc:407] Connection header received
0621 02:55:13.445953 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.445959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.446034 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.446136 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:13.447477 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.448515 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.450916 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.451291 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.451388 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.452918 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:13.452922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:13.452925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:13.452941 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:13.452969 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:13.452980 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:13.453051 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.453325 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.453398 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 02:55:13.455026 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.441808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44675 (local address 127.11.109.2:46661)
0621 02:55:13.442931 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:13.442989 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:13.445664 (+ 2675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:13.446378 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:13.446391 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:13.446503 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:13.447297 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.447322 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.448669 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.448677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.450780 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.450798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.451523 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.451530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.451849 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.452585 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:55:13.452591 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:13.452642 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:55:13.452733 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:13.453075 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:13.453088 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:13.453099 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:55:13.453309 (+ 210us) client_negotiation.cc:240] Negotiation successful
0621 02:55:13.453710 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":345,"threads_started":1}
I20260621 02:55:13.456476 31895 heartbeater.cc:347] Connected to a master server at 127.11.109.62:44675
I20260621 02:55:13.456789 31895 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:13.457662 31895 heartbeater.cc:511] Master 127.11.109.62:44675 requested a full tablet report, sending...
I20260621 02:55:13.459167 31578 ts_manager.cc:194] Registered new tserver with Master: f80ad02d33f14fd993b88499b79e6025 (127.11.109.2:46749)
I20260621 02:55:13.459968 31578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.2:46661
W20260621 02:55:13.762271 31898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:13.762521 31898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:13.762689 31898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:13.769740 31898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:13.769825 31898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:13.770009 31898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.109.3
I20260621 02:55:13.778947 31898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.109.3
--webserver_port=0
--tserver_master_addrs=127.11.109.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.109.3
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:13.780627 31898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:13.783062 31898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:13.797595 31903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:13.797566 31904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:13.797631 31906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:13.798131 31898 server_base.cc:1034] running on GCE node
I20260621 02:55:13.798799 31898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:13.799676 31898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:13.800869 31898 hybrid_clock.cc:648] HybridClock initialized: now 1782010513800825 us; error 43 us; skew 500 ppm
I20260621 02:55:13.801219 31898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:13.803328 31898 webserver.cc:460] Webserver started at http://127.11.109.3:34923/ using document root <none> and password file <none>
I20260621 02:55:13.803990 31898 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:13.804100 31898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:13.804414 31898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:13.807195 31898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data/instance:
uuid: "50fdc380393e4f1ea2c9913663edfc92"
format_stamp: "Formatted at 2026-06-21 02:55:13 on dist-test-slave-bc52"
server_key: "eff17d1944a605e8ba51f8c6752907f5"
server_key_iv: "9531080fa1f9a4d8fd4399ea04c6120d"
server_key_version: "encryptionkey@0"
I20260621 02:55:13.807955 31898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance:
uuid: "50fdc380393e4f1ea2c9913663edfc92"
format_stamp: "Formatted at 2026-06-21 02:55:13 on dist-test-slave-bc52"
server_key: "eff17d1944a605e8ba51f8c6752907f5"
server_key_iv: "9531080fa1f9a4d8fd4399ea04c6120d"
server_key_version: "encryptionkey@0"
I20260621 02:55:13.812749 31898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260621 02:55:13.815997 31912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:13.817308 31898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:13.817473 31898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal
uuid: "50fdc380393e4f1ea2c9913663edfc92"
format_stamp: "Formatted at 2026-06-21 02:55:13 on dist-test-slave-bc52"
server_key: "eff17d1944a605e8ba51f8c6752907f5"
server_key_iv: "9531080fa1f9a4d8fd4399ea04c6120d"
server_key_version: "encryptionkey@0"
I20260621 02:55:13.817680 31898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:13.847060 31898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:13.847865 31898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:13.848160 31898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:13.849098 31898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:55:13.850538 31898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:55:13.850654 31898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:13.850787 31898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:55:13.850872 31898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:13.882736 31898 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.3:37759
I20260621 02:55:13.882823 32024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.3:37759 every 8 connection(s)
I20260621 02:55:13.884021 31898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/data/info.pb
I20260621 02:55:13.891877 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 31898
I20260621 02:55:13.892069 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal/instance
I20260621 02:55:13.892519 11700 external_mini_cluster.cc:1428] Setting key c5db57336e8c2fc2907bd2ec5f032ddf
I20260621 02:55:13.896687 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.886556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.3:45131 (local address 127.11.109.62:44675)
0621 02:55:13.886734 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:13.886738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.888055 (+ 1317us) server_negotiation.cc:407] Connection header received
0621 02:55:13.889878 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.889883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.889971 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.890088 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:13.891101 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.892137 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.893934 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.894475 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.894573 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.896055 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:13.896059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:13.896062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:13.896078 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:13.896106 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:13.896118 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:13.896197 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.896377 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.896444 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:55:13.897773 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.887054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:44675 (local address 127.11.109.3:45131)
0621 02:55:13.887896 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:13.887936 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:13.889612 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:13.890309 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:13.890317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:13.890397 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:13.890916 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.890934 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.892293 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.892298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.893805 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.893817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.894690 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.894697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.894999 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.895761 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:55:13.895764 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:13.895823 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:55:13.895895 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:13.896211 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:13.896218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:13.896223 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:55:13.896355 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 02:55:13.896660 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":263,"threads_started":1}
I20260621 02:55:13.899118 32025 heartbeater.cc:347] Connected to a master server at 127.11.109.62:44675
I20260621 02:55:13.899384 32025 heartbeater.cc:464] Registering TS with master...
I20260621 02:55:13.900049 32025 heartbeater.cc:511] Master 127.11.109.62:44675 requested a full tablet report, sending...
I20260621 02:55:13.901397 31578 ts_manager.cc:194] Registered new tserver with Master: 50fdc380393e4f1ea2c9913663edfc92 (127.11.109.3:37759)
I20260621 02:55:13.902215 31578 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.109.3:45131
I20260621 02:55:13.907145 11700 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:55:13.916306 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.910955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.11.109.62:44675)
0621 02:55:13.911131 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:13.911134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.911597 (+ 463us) server_negotiation.cc:407] Connection header received
0621 02:55:13.911727 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.911731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.911793 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.911867 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:13.912269 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.912929 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.914373 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.914787 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.914876 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.915820 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:13.915824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:13.915827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:13.915841 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:13.915865 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:13.915875 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:13.915936 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.916058 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.916117 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:13.919795 31578 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:55:13.926503 31578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:55:13.966969 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.937647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34474)
0621 02:55:13.958647 (+ 21000us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:13.958674 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:13.958852 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:13.959632 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:13.959640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:13.959668 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:13.960060 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.960071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.961794 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.961801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.963984 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.964002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.965961 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.965967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.966177 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.966234 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 02:55:13.966367 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 02:55:13.966572 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20831,"mutex_wait_us":6,"thread_start_us":21069,"threads_started":1}
I20260621 02:55:13.967289 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.937626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.11.109.3:37759)
0621 02:55:13.959363 (+ 21737us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:13.959371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.959397 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:55:13.959490 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.959496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.959667 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.960613 (+ 946us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:13.960896 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.962405 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.964801 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.966218 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.966376 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.966640 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.966784 (+ 144us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.966947 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21559,"thread_start_us":21379,"threads_started":1}
I20260621 02:55:13.967867 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.937396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:37759 (local address 127.0.0.1:38802)
0621 02:55:13.953107 (+ 15711us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:13.953159 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:13.954032 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:13.959864 (+ 5832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:13.959869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:13.959893 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:13.960384 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.960399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.962571 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.962577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.964579 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.964593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.966423 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.966430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.966616 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.966649 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:55:13.966733 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 02:55:13.966858 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15409,"mutex_wait_us":151,"thread_start_us":15262,"threads_started":1}
I20260621 02:55:13.968056 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.938014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:46749 (local address 127.0.0.1:49600)
0621 02:55:13.958997 (+ 20983us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:13.959017 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:13.959145 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:13.960084 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:13.960088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:13.960109 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:13.960586 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.960596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.962562 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.962568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.965189 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:13.965206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.967288 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.967295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:13.967514 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.967547 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:55:13.967635 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:55:13.967751 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20847,"mutex_wait_us":349,"thread_start_us":20297,"threads_started":1}
I20260621 02:55:13.968238 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.939112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.11.109.2:46749)
0621 02:55:13.958132 (+ 19020us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:13.958145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.959269 (+ 1124us) server_negotiation.cc:407] Connection header received
0621 02:55:13.959377 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.959387 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.959616 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.959822 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:13.960779 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.962381 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.965405 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.966782 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.966916 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.967157 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.967714 (+ 557us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.967898 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18807,"thread_start_us":18684,"threads_started":1}
I20260621 02:55:13.968681 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:13.938630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.11.109.1:45725)
0621 02:55:13.956288 (+ 17658us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:13.956301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:13.959032 (+ 2731us) server_negotiation.cc:407] Connection header received
0621 02:55:13.959150 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:13.959159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:13.959410 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:13.959617 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:13.960327 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.961661 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.964208 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:13.965762 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:13.965898 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:13.966147 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:13.967078 (+ 931us) server_negotiation.cc:299] Negotiation successful
0621 02:55:13.968446 (+ 1368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17429,"thread_start_us":17327,"threads_started":1}
I20260621 02:55:13.971992 31699 tablet_service.cc:1467] Processing CreateTablet for tablet 77a4f4fc03cc4a8cbfc185858a8f539c (DEFAULT_TABLE table=test-table [id=20f477648fb94324a17941ac51e3053b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:13.971992 31830 tablet_service.cc:1467] Processing CreateTablet for tablet 77a4f4fc03cc4a8cbfc185858a8f539c (DEFAULT_TABLE table=test-table [id=20f477648fb94324a17941ac51e3053b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:13.971978 31960 tablet_service.cc:1467] Processing CreateTablet for tablet 77a4f4fc03cc4a8cbfc185858a8f539c (DEFAULT_TABLE table=test-table [id=20f477648fb94324a17941ac51e3053b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:55:13.973296 31960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a4f4fc03cc4a8cbfc185858a8f539c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:13.973305 31699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a4f4fc03cc4a8cbfc185858a8f539c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:13.973305 31830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a4f4fc03cc4a8cbfc185858a8f539c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:13.984719 32043 tablet_bootstrap.cc:492] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Bootstrap starting.
I20260621 02:55:13.987351 32045 tablet_bootstrap.cc:492] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Bootstrap starting.
I20260621 02:55:13.988111 32044 tablet_bootstrap.cc:492] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Bootstrap starting.
I20260621 02:55:13.988613 32043 tablet_bootstrap.cc:654] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:13.989930 32043 log.cc:824] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:13.991314 32045 tablet_bootstrap.cc:654] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:13.991612 32044 tablet_bootstrap.cc:654] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:13.992446 32045 log.cc:824] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:13.992718 32044 log.cc:824] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:13.994205 32043 tablet_bootstrap.cc:492] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: No bootstrap required, opened a new log
I20260621 02:55:13.994554 32043 ts_tablet_manager.cc:1397] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 02:55:13.995916 32044 tablet_bootstrap.cc:492] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: No bootstrap required, opened a new log
I20260621 02:55:13.996229 32044 ts_tablet_manager.cc:1397] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:55:14.006022 32045 tablet_bootstrap.cc:492] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: No bootstrap required, opened a new log
I20260621 02:55:14.006335 32045 ts_tablet_manager.cc:1397] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.010s
I20260621 02:55:14.006469 32043 raft_consensus.cc:348] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.006888 32043 raft_consensus.cc:374] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:14.006609 32044 raft_consensus.cc:348] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.007038 32044 raft_consensus.cc:374] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:14.007036 32043 raft_consensus.cc:729] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fdc380393e4f1ea2c9913663edfc92, State: Initialized, Role: FOLLOWER
I20260621 02:55:14.007148 32044 raft_consensus.cc:729] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80ad02d33f14fd993b88499b79e6025, State: Initialized, Role: FOLLOWER
I20260621 02:55:14.007709 32043 consensus_queue.cc:260] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.007876 32044 consensus_queue.cc:260] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.014876 32025 heartbeater.cc:503] Master 127.11.109.62:44675 was elected leader, sending a full tablet report...
I20260621 02:55:14.015954 32043 ts_tablet_manager.cc:1428] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Time spent starting tablet: real 0.021s user 0.012s sys 0.006s
I20260621 02:55:14.017277 32044 ts_tablet_manager.cc:1428] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20260621 02:55:14.017498 31895 heartbeater.cc:503] Master 127.11.109.62:44675 was elected leader, sending a full tablet report...
I20260621 02:55:14.018476 32045 raft_consensus.cc:348] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.018983 32045 raft_consensus.cc:374] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:14.019109 32045 raft_consensus.cc:729] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3171452ca8db46f1a725856535340da5, State: Initialized, Role: FOLLOWER
I20260621 02:55:14.019698 32045 consensus_queue.cc:260] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.020679 31764 heartbeater.cc:503] Master 127.11.109.62:44675 was elected leader, sending a full tablet report...
I20260621 02:55:14.021384 32045 ts_tablet_manager.cc:1428] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260621 02:55:14.032939 32051 raft_consensus.cc:482] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:55:14.033228 32051 raft_consensus.cc:504] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.034675 32051 leader_election.cc:288] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fdc380393e4f1ea2c9913663edfc92 (127.11.109.3:37759), f80ad02d33f14fd993b88499b79e6025 (127.11.109.2:46749)
I20260621 02:55:14.043396 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.035082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:38963 (local address 127.11.109.3:37759)
0621 02:55:14.035275 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:14.035284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:14.035757 (+ 473us) server_negotiation.cc:407] Connection header received
0621 02:55:14.035979 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:14.036002 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:14.036079 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:14.036176 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:14.036974 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.038057 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.041042 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.042266 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.042412 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.042478 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:14.043038 (+ 560us) server_negotiation.cc:299] Negotiation successful
0621 02:55:14.043152 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 02:55:14.043401 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.035136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.109.1:41787 (local address 127.11.109.2:46749)
0621 02:55:14.035338 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:14.035345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:14.035901 (+ 556us) server_negotiation.cc:407] Connection header received
0621 02:55:14.036062 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:14.036068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:14.036157 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:14.036240 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:55:14.036950 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.038055 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.041043 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.042245 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.042408 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.042478 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:14.043026 (+ 548us) server_negotiation.cc:299] Negotiation successful
0621 02:55:14.043150 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:55:14.043606 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.035031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.2:46749 (local address 127.11.109.1:41787)
0621 02:55:14.035822 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.035836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:14.035937 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:14.036257 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:14.036261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:14.036286 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:14.036758 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:14.036776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.038190 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.038194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:14.040818 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:14.040833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.042460 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.042468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:14.042799 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.042867 (+ 68us) client_negotiation.cc:769] Sending connection context
0621 02:55:14.042961 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:55:14.043180 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":372,"thread_start_us":476,"threads_started":1}
I20260621 02:55:14.043651 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.034904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.3:37759 (local address 127.11.109.1:38963)
0621 02:55:14.035630 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.035660 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:14.035810 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:14.036194 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:14.036198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:14.036243 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:55:14.036839 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:14.036852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.038189 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.038194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:14.040824 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:14.040833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.042477 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.042482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:14.042794 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.042869 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 02:55:14.042961 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:55:14.043180 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":279,"thread_start_us":376,"threads_started":1}
I20260621 02:55:14.044574 31850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a4f4fc03cc4a8cbfc185858a8f539c" candidate_uuid: "3171452ca8db46f1a725856535340da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f80ad02d33f14fd993b88499b79e6025" is_pre_election: true
I20260621 02:55:14.044636 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a4f4fc03cc4a8cbfc185858a8f539c" candidate_uuid: "3171452ca8db46f1a725856535340da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fdc380393e4f1ea2c9913663edfc92" is_pre_election: true
I20260621 02:55:14.045071 31850 raft_consensus.cc:2454] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3171452ca8db46f1a725856535340da5 in term 0.
I20260621 02:55:14.045104 31980 raft_consensus.cc:2454] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3171452ca8db46f1a725856535340da5 in term 0.
I20260621 02:55:14.045848 31655 leader_election.cc:302] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3171452ca8db46f1a725856535340da5, 50fdc380393e4f1ea2c9913663edfc92, f80ad02d33f14fd993b88499b79e6025; no voters:
I20260621 02:55:14.046298 32051 raft_consensus.cc:2781] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:55:14.046443 32051 raft_consensus.cc:482] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:55:14.046566 32051 raft_consensus.cc:3037] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:14.048652 32051 raft_consensus.cc:504] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.049264 32051 leader_election.cc:288] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [CANDIDATE]: Term 1 election: Requested vote from peers 50fdc380393e4f1ea2c9913663edfc92 (127.11.109.3:37759), f80ad02d33f14fd993b88499b79e6025 (127.11.109.2:46749)
I20260621 02:55:14.049643 31980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a4f4fc03cc4a8cbfc185858a8f539c" candidate_uuid: "3171452ca8db46f1a725856535340da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fdc380393e4f1ea2c9913663edfc92"
I20260621 02:55:14.049652 31850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a4f4fc03cc4a8cbfc185858a8f539c" candidate_uuid: "3171452ca8db46f1a725856535340da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f80ad02d33f14fd993b88499b79e6025"
I20260621 02:55:14.049834 31850 raft_consensus.cc:3037] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:14.049815 31980 raft_consensus.cc:3037] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:14.052296 31980 raft_consensus.cc:2454] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3171452ca8db46f1a725856535340da5 in term 1.
I20260621 02:55:14.052441 31850 raft_consensus.cc:2454] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3171452ca8db46f1a725856535340da5 in term 1.
I20260621 02:55:14.052695 31655 leader_election.cc:302] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3171452ca8db46f1a725856535340da5, 50fdc380393e4f1ea2c9913663edfc92; no voters:
I20260621 02:55:14.052965 32051 raft_consensus.cc:2781] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:14.053434 32051 raft_consensus.cc:686] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 1 LEADER]: Becoming Leader. State: Replica: 3171452ca8db46f1a725856535340da5, State: Running, Role: LEADER
I20260621 02:55:14.053917 32051 consensus_queue.cc:237] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } }
I20260621 02:55:14.057853 31578 catalog_manager.cc:5495] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3171452ca8db46f1a725856535340da5 (127.11.109.1). New cstate: current_term: 1 leader_uuid: "3171452ca8db46f1a725856535340da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3171452ca8db46f1a725856535340da5" member_type: VOTER last_known_addr { host: "127.11.109.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 } health_report { overall_health: UNKNOWN } } }
I20260621 02:55:14.085840 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.081049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.11.109.1:45725)
0621 02:55:14.081273 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:14.081278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:14.081358 (+ 80us) server_negotiation.cc:407] Connection header received
0621 02:55:14.081512 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:14.081515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:14.081609 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:14.081742 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:14.082035 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.082717 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.083972 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.084355 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.084450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.084870 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:14.085389 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:14.085499 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:14.085584 (+ 85us) server_negotiation.cc:299] Negotiation successful
0621 02:55:14.085650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:55:14.093405 31850 raft_consensus.cc:1261] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 1 FOLLOWER]: Refusing update from remote peer 3171452ca8db46f1a725856535340da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:14.093497 31980 raft_consensus.cc:1261] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 1 FOLLOWER]: Refusing update from remote peer 3171452ca8db46f1a725856535340da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:14.094357 32054 consensus_queue.cc:1035] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80ad02d33f14fd993b88499b79e6025" member_type: VOTER last_known_addr { host: "127.11.109.2" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:14.094692 32051 consensus_queue.cc:1035] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fdc380393e4f1ea2c9913663edfc92" member_type: VOTER last_known_addr { host: "127.11.109.3" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:14.110831 32058 mvcc.cc:205] Tried to move back new op lower bound from 7299115065717366784 to 7299115065568178176. Current Snapshot: MvccSnapshot[applied={T|T < 7299115065717366784}]
I20260621 02:55:14.117362 32059 mvcc.cc:205] Tried to move back new op lower bound from 7299115065717366784 to 7299115065568178176. Current Snapshot: MvccSnapshot[applied={T|T < 7299115065717366784}]
I20260621 02:55:14.121852 31576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:14.122138 31576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:55:14.126031 31576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74e4998b1a914589ac62348237061adc: Sending DeleteTablet for 3 replicas of tablet 77a4f4fc03cc4a8cbfc185858a8f539c
I20260621 02:55:14.127229 31699 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a4f4fc03cc4a8cbfc185858a8f539c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:14 UTC) from {username='slave'} at 127.0.0.1:34474
I20260621 02:55:14.127271 31830 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a4f4fc03cc4a8cbfc185858a8f539c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:14 UTC) from {username='slave'} at 127.0.0.1:49600
I20260621 02:55:14.127960 31960 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a4f4fc03cc4a8cbfc185858a8f539c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:55:14 UTC) from {username='slave'} at 127.0.0.1:38802
I20260621 02:55:14.128672 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31637
I20260621 02:55:14.133745 32069 tablet_replica.cc:330] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: stopping tablet replica
I20260621 02:55:14.134232 32069 raft_consensus.cc:2229] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:14.134789 32069 raft_consensus.cc:2258] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:14.135639 32068 tablet_replica.cc:330] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: stopping tablet replica
I20260621 02:55:14.136096 32068 raft_consensus.cc:2229] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:55:14.136792 32068 raft_consensus.cc:2258] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:14.142498 32069 ts_tablet_manager.cc:1905] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:14.143865 32068 ts_tablet_manager.cc:1905] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:14.149708 32067 tablet_replica.cc:330] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: stopping tablet replica
I20260621 02:55:14.150153 32067 raft_consensus.cc:2229] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:55:14.150695 32067 raft_consensus.cc:2258] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:55:14.153219 32067 ts_tablet_manager.cc:1905] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:55:14.153753 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.129955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.11.109.1:45725)
0621 02:55:14.130206 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:14.130215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:14.137558 (+ 7343us) server_negotiation.cc:407] Connection header received
0621 02:55:14.137748 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:14.137756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:14.137881 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:14.138027 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:14.138782 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.140339 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.149788 (+ 9449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.150468 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.150630 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.152021 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:14.152031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:14.152046 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:14.152801 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:14.152858 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:14.152881 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:14.152958 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:14.153242 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 02:55:14.153348 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:55:14.153427 32068 ts_tablet_manager.cc:1918] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:14.154488 32068 log.cc:1196] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-0/wal/wals/77a4f4fc03cc4a8cbfc185858a8f539c
I20260621 02:55:14.154511 31759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:14.151265 32069 ts_tablet_manager.cc:1918] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:14.155200 32069 log.cc:1196] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-2/wal/wals/77a4f4fc03cc4a8cbfc185858a8f539c
I20260621 02:55:14.155913 32069 ts_tablet_manager.cc:1939] T 77a4f4fc03cc4a8cbfc185858a8f539c P 50fdc380393e4f1ea2c9913663edfc92: Deleting consensus metadata
I20260621 02:55:14.158314 31565 catalog_manager.cc:4841] TS 50fdc380393e4f1ea2c9913663edfc92 (127.11.109.3:37759): tablet 77a4f4fc03cc4a8cbfc185858a8f539c (table test-table [id=20f477648fb94324a17941ac51e3053b]) successfully deleted
I20260621 02:55:14.159569 32067 ts_tablet_manager.cc:1918] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:55:14.159729 32067 log.cc:1196] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Deleting WAL directory at /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782010204918124-11700-0/minicluster-data/ts-1/wal/wals/77a4f4fc03cc4a8cbfc185858a8f539c
I20260621 02:55:14.160192 32067 ts_tablet_manager.cc:1939] T 77a4f4fc03cc4a8cbfc185858a8f539c P f80ad02d33f14fd993b88499b79e6025: Deleting consensus metadata
I20260621 02:55:14.161648 31563 catalog_manager.cc:4841] TS f80ad02d33f14fd993b88499b79e6025 (127.11.109.2:46749): tablet 77a4f4fc03cc4a8cbfc185858a8f539c (table test-table [id=20f477648fb94324a17941ac51e3053b]) successfully deleted
I20260621 02:55:14.250458 32068 ts_tablet_manager.cc:1939] T 77a4f4fc03cc4a8cbfc185858a8f539c P 3171452ca8db46f1a725856535340da5: Deleting consensus metadata
I20260621 02:55:14.250641 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31637
W20260621 02:55:14.269714 31564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.109.1:45725 (error 108)
I20260621 02:55:14.270437 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31768
I20260621 02:55:14.271656 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.271135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34498)
0621 02:55:14.271424 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.271527 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260621 02:55:14.271875 31564 catalog_manager.cc:4584] TS 3171452ca8db46f1a725856535340da5 (127.11.109.1:45725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77a4f4fc03cc4a8cbfc185858a8f539c: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
I20260621 02:55:14.277292 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.271268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.11.109.2:46749)
0621 02:55:14.271450 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:14.271458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:14.271476 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:55:14.271685 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:14.271691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:14.271820 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:14.271918 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:14.272387 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.273148 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.274420 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.274821 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.274945 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.275915 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:14.275922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:14.275935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:14.276510 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:14.276555 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:14.276573 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:14.276701 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:14.276824 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 02:55:14.276924 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:55:14.277931 31890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:14.325668 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.325215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34504)
0621 02:55:14.325407 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.325505 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:55:14.326725 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.326330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34520)
0621 02:55:14.326472 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.326554 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 02:55:14.366532 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31768
I20260621 02:55:14.383541 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31898
I20260621 02:55:14.387012 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.386542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34532)
0621 02:55:14.386764 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.386840 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 02:55:14.387979 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.387651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34548)
0621 02:55:14.387787 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.387863 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 02:55:14.390645 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.384253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.11.109.3:37759)
0621 02:55:14.384448 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:14.384454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:14.384563 (+ 109us) server_negotiation.cc:407] Connection header received
0621 02:55:14.384783 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:14.384788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:14.384901 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:14.385057 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:14.385481 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.386207 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.387533 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:14.387937 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:14.388097 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:14.389302 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:14.389315 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:14.389336 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:14.389937 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:14.389981 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:14.390000 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:14.390145 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:14.390244 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 02:55:14.390346 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:55:14.391238 32020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:14.468679 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.468217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34554)
0621 02:55:14.468428 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.468544 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 02:55:14.469856 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:14.469418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.1:45725 (local address 127.0.0.1:34560)
0621 02:55:14.469653 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:14.469750 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.1:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 02:55:14.494230 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31898
I20260621 02:55:14.511113 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 31547
I20260621 02:55:14.512166 31608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:14.620540 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 31547
2026-06-21T02:55:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2670 ms)
[----------] 6 tests from EncryptionPolicyTest (20949 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:14 dist-test-slave-bc52 krb5kdc[32078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:14 dist-test-slave-bc52 krb5kdc[32078](info): set up 2 sockets
Jun 21 02:55:14 dist-test-slave-bc52 krb5kdc[32078](info): commencing operation
krb5kdc: starting...
W20260621 02:55:15.709650 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:15 dist-test-slave-bc52 krb5kdc[32078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:15.943980 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:16.262908 32091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:16.263224 32091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:16.263398 32091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:16.270587 32091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:16.270742 32091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:16.270848 32091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:16.270967 32091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:16.271041 32091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:16.280663 32091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:16.281965 32091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:16.283234 32091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:16.295120 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:16.295150 32097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:16.295081 32099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:16.295774 32091 server_base.cc:1034] running on GCE node
I20260621 02:55:16.296509 32091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:16.297974 32091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:16.299232 32091 hybrid_clock.cc:648] HybridClock initialized: now 1782010516299187 us; error 52 us; skew 500 ppm
I20260621 02:55:16.299600 32091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:16 dist-test-slave-bc52 krb5kdc[32078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:16.305896 32091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:16.307392 32091 webserver.cc:460] Webserver started at http://127.0.0.1:35905/ using document root <none> and password file <none>
I20260621 02:55:16.308040 32091 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:16.308195 32091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:16.308501 32091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:16.311436 32091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "615599b45cac4522af91abeaa3f33245"
format_stamp: "Formatted at 2026-06-21 02:55:16 on dist-test-slave-bc52"
server_key: "cc7c4799f226cc6e7323f2834f6462be"
server_key_iv: "67b85b86bbce9117d7e7656dc2a5d109"
server_key_version: "encryptionkey@0"
I20260621 02:55:16.312197 32091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "615599b45cac4522af91abeaa3f33245"
format_stamp: "Formatted at 2026-06-21 02:55:16 on dist-test-slave-bc52"
server_key: "cc7c4799f226cc6e7323f2834f6462be"
server_key_iv: "67b85b86bbce9117d7e7656dc2a5d109"
server_key_version: "encryptionkey@0"
I20260621 02:55:16.316992 32091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260621 02:55:16.320245 32106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:16.321578 32091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:16.321749 32091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "615599b45cac4522af91abeaa3f33245"
format_stamp: "Formatted at 2026-06-21 02:55:16 on dist-test-slave-bc52"
server_key: "cc7c4799f226cc6e7323f2834f6462be"
server_key_iv: "67b85b86bbce9117d7e7656dc2a5d109"
server_key_version: "encryptionkey@0"
I20260621 02:55:16.321929 32091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:16.340006 32091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:16.344491 32091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:16.344795 32091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:16.362762 32091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42815
I20260621 02:55:16.362787 32157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42815 every 8 connection(s)
I20260621 02:55:16.364090 32091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:16.365456 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32091
I20260621 02:55:16.365675 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:16.366464 11700 external_mini_cluster.cc:1428] Setting key e6566db3d80ce6445909d8a9654e4894
I20260621 02:55:16.368655 32158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:55:16 dist-test-slave-bc52 krb5kdc[32078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:55:16.381788 32158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245: Bootstrap starting.
I20260621 02:55:16.384763 32158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:16.386139 32158 log.cc:824] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:16.388990 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:16.368345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42799 (local address 127.0.0.1:42815)
0621 02:55:16.369093 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:16.369103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:16.369143 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:55:16.370917 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:16.370937 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:16.371306 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:16.371616 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:16.373373 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:16.374643 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:16.376425 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:16.377048 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:16.377231 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:16.381847 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:16.381854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:16.381867 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:16.381900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:16.384822 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:16.385817 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:16.385822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:16.385828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:16.385972 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:16.386424 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:16.386427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:16.386429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:16.387103 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:16.387458 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:16.387805 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 02:55:16.388094 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":353,"threads_started":1}
I20260621 02:55:16.389071 32158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245: No bootstrap required, opened a new log
I20260621 02:55:16.392657 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:16.398663 32158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615599b45cac4522af91abeaa3f33245" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260621 02:55:16.399101 32158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:16.399230 32158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615599b45cac4522af91abeaa3f33245, State: Initialized, Role: FOLLOWER
I20260621 02:55:16.399745 32158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615599b45cac4522af91abeaa3f33245" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260621 02:55:16.399964 32158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:16.400096 32158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:16.400264 32158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:16.402952 32158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615599b45cac4522af91abeaa3f33245" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260621 02:55:16.403425 32158 leader_election.cc:302] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615599b45cac4522af91abeaa3f33245; no voters:
I20260621 02:55:16.404064 32158 leader_election.cc:288] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:16.404155 32168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:16.404814 32168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [term 1 LEADER]: Becoming Leader. State: Replica: 615599b45cac4522af91abeaa3f33245, State: Running, Role: LEADER
I20260621 02:55:16.405589 32168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615599b45cac4522af91abeaa3f33245" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260621 02:55:16.408246 32158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:16.409009 32169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615599b45cac4522af91abeaa3f33245. Latest consensus state: current_term: 1 leader_uuid: "615599b45cac4522af91abeaa3f33245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615599b45cac4522af91abeaa3f33245" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } } }
I20260621 02:55:16.409446 32169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:16.410429 32170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615599b45cac4522af91abeaa3f33245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615599b45cac4522af91abeaa3f33245" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } } }
I20260621 02:55:16.415108 32170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:16.417238 32173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:16.418409 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:16.398090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56481 (local address 127.0.0.1:42815)
0621 02:55:16.398255 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:16.398260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:16.398764 (+ 504us) server_negotiation.cc:407] Connection header received
0621 02:55:16.398869 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:16.398875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:16.398945 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:16.399052 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:16.400685 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:16.401940 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:16.403238 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:16.403728 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:16.403850 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:16.406519 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:16.406529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:16.406534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:16.406572 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:16.410814 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:16.411540 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:16.411545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:16.411549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:16.415111 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:16.415809 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:16.415816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:16.415822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:16.416195 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:16.416398 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:16.417618 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0621 02:55:16.417877 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:55:16.424361 32173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:16.432962 32173 catalog_manager.cc:1347] Generated new cluster ID: 37393e811fc548e996abf1147abd9e8f
I20260621 02:55:16.433094 32173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:16.442950 32173 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:16.444154 32173 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:16.449344 32173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615599b45cac4522af91abeaa3f33245: Generated new TSK 0
I20260621 02:55:16.450314 32173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:16.456097 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32091
I20260621 02:55:16.457604 32153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:16.544314 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32091
W20260621 02:55:16.561340 32076 connection.cc:535] client connection to 127.0.0.1:42815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42815): Connection reset by peer (error 104)
2026-06-21T02:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:16 dist-test-slave-bc52 krb5kdc[32190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:16 dist-test-slave-bc52 krb5kdc[32190](info): set up 2 sockets
Jun 21 02:55:16 dist-test-slave-bc52 krb5kdc[32190](info): commencing operation
krb5kdc: starting...
W20260621 02:55:17.637665 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:17 dist-test-slave-bc52 krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:17.874519 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:18.190721 32203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:18.190990 32203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:18.191123 32203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:18.198099 32203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:18.198199 32203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:18.198312 32203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:18.198413 32203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:18.198493 32203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:18.207924 32203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:18.209062 32203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:18.210309 32203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:18.221758 32208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:18.221796 32209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:18.221709 32211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:18.222450 32203 server_base.cc:1034] running on GCE node
I20260621 02:55:18.223176 32203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:18.224455 32203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:18.225642 32203 hybrid_clock.cc:648] HybridClock initialized: now 1782010518225593 us; error 50 us; skew 500 ppm
I20260621 02:55:18.225924 32203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:18 dist-test-slave-bc52 krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:18.228906 32203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:18.230355 32203 webserver.cc:460] Webserver started at http://127.0.0.1:37761/ using document root <none> and password file <none>
I20260621 02:55:18.230932 32203 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:18.231027 32203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:18.231269 32203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:18.234055 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "004b8276c0644c2398ccb136cf02fc51"
format_stamp: "Formatted at 2026-06-21 02:55:18 on dist-test-slave-bc52"
server_key: "71f936d1677b05bf9af0428195da3392"
server_key_iv: "79c714f55f3215a52a25e3e018bf7b99"
server_key_version: "encryptionkey@0"
I20260621 02:55:18.234738 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "004b8276c0644c2398ccb136cf02fc51"
format_stamp: "Formatted at 2026-06-21 02:55:18 on dist-test-slave-bc52"
server_key: "71f936d1677b05bf9af0428195da3392"
server_key_iv: "79c714f55f3215a52a25e3e018bf7b99"
server_key_version: "encryptionkey@0"
I20260621 02:55:18.239528 32203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:55:18.242640 32218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:18.243809 32203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:55:18.243930 32203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "004b8276c0644c2398ccb136cf02fc51"
format_stamp: "Formatted at 2026-06-21 02:55:18 on dist-test-slave-bc52"
server_key: "71f936d1677b05bf9af0428195da3392"
server_key_iv: "79c714f55f3215a52a25e3e018bf7b99"
server_key_version: "encryptionkey@0"
I20260621 02:55:18.244072 32203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:18.291035 32203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:18.295342 32203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:18.295570 32203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:18.313695 32203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36321
I20260621 02:55:18.313753 32269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36321 every 8 connection(s)
I20260621 02:55:18.314992 32203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:18.317873 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32203
I20260621 02:55:18.318058 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:18.318619 11700 external_mini_cluster.cc:1428] Setting key 5bd31cfb4d512f95b0da68abbff019b8
I20260621 02:55:18.318842 32270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:18.333293 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:18.320551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45805 (local address 127.0.0.1:36321)
0621 02:55:18.321487 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:18.321524 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:18.321915 (+ 391us) server_negotiation.cc:407] Connection header received
0621 02:55:18.323491 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:18.323518 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:18.324017 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:18.324434 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:18.324974 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:18.326978 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:18.328323 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:18.329390 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:18.329591 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:18.330659 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:18.330670 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:18.330692 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:18.331383 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:18.331501 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:18.331532 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:18.331685 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:18.331930 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 02:55:18.332084 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":394,"threads_started":1}
I20260621 02:55:18.333389 32270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51: Bootstrap starting.
I20260621 02:55:18.336408 32270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:18.336824 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:18.337595 32270 log.cc:824] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:18.340700 32270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51: No bootstrap required, opened a new log
I20260621 02:55:18.348999 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:18.341753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40387 (local address 127.0.0.1:36321)
0621 02:55:18.341915 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:18.341922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:18.342336 (+ 414us) server_negotiation.cc:407] Connection header received
0621 02:55:18.342462 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:18.342467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:18.342541 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:18.342650 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:18.343082 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:18.344249 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:18.346182 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:18.346996 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:18.347138 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:18.348252 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:18.348258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:18.348263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:18.348280 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:18.348321 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:18.348339 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:18.348445 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:18.348631 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 02:55:18.348679 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 02:55:18.350981 32270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b8276c0644c2398ccb136cf02fc51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36321 } }
I20260621 02:55:18.351310 32270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:18.351382 32270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004b8276c0644c2398ccb136cf02fc51, State: Initialized, Role: FOLLOWER
I20260621 02:55:18.351835 32270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b8276c0644c2398ccb136cf02fc51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36321 } }
I20260621 02:55:18.352030 32270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:18.352129 32270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:18.352244 32270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:18.354362 32270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b8276c0644c2398ccb136cf02fc51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36321 } }
I20260621 02:55:18.354780 32270 leader_election.cc:302] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 004b8276c0644c2398ccb136cf02fc51; no voters:
I20260621 02:55:18.355244 32270 leader_election.cc:288] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:18.355407 32280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:18.356930 32280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [term 1 LEADER]: Becoming Leader. State: Replica: 004b8276c0644c2398ccb136cf02fc51, State: Running, Role: LEADER
I20260621 02:55:18.357487 32270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:18.357559 32280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b8276c0644c2398ccb136cf02fc51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36321 } }
I20260621 02:55:18.360584 32281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "004b8276c0644c2398ccb136cf02fc51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b8276c0644c2398ccb136cf02fc51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36321 } } }
I20260621 02:55:18.360942 32281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:18.360833 32282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004b8276c0644c2398ccb136cf02fc51. Latest consensus state: current_term: 1 leader_uuid: "004b8276c0644c2398ccb136cf02fc51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004b8276c0644c2398ccb136cf02fc51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36321 } } }
I20260621 02:55:18.361224 32282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:18.364660 32287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:18.369724 32287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:18.378993 32287 catalog_manager.cc:1347] Generated new cluster ID: 9ccf90dfc33f4b4f9bce94a1e2cfc76a
I20260621 02:55:18.379108 32287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:18.402626 32287 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:18.404349 32287 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:18.418416 32287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 004b8276c0644c2398ccb136cf02fc51: Generated new TSK 0
I20260621 02:55:18.419382 32287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:18.431635 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32203
W20260621 02:55:18.432714 32265 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45805
W20260621 02:55:18.433449 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:18.433633 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32203
W20260621 02:55:18.450506 32188 connection.cc:535] client connection to 127.0.0.1:36321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36321): Connection reset by peer (error 104)
2026-06-21T02:55:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:18 dist-test-slave-bc52 krb5kdc[32301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:18 dist-test-slave-bc52 krb5kdc[32301](info): set up 2 sockets
Jun 21 02:55:18 dist-test-slave-bc52 krb5kdc[32301](info): commencing operation
krb5kdc: starting...
W20260621 02:55:19.533531 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:19 dist-test-slave-bc52 krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:19.762197 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:20.078202 32314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:20.078547 32314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:20.078724 32314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:20.085788 32314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:20.085937 32314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:20.086050 32314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:20.086177 32314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:20.086274 32314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:20.095969 32314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:20.097330 32314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:20.098659 32314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:20.110181 32320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:20.110210 32319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:20.110675 32314 server_base.cc:1034] running on GCE node
W20260621 02:55:20.110764 32322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:20.111555 32314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:20.112866 32314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:20.114058 32314 hybrid_clock.cc:648] HybridClock initialized: now 1782010520113985 us; error 72 us; skew 500 ppm
I20260621 02:55:20.114413 32314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:20 dist-test-slave-bc52 krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:20.117389 32314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:20.118753 32314 webserver.cc:460] Webserver started at http://127.0.0.1:35383/ using document root <none> and password file <none>
I20260621 02:55:20.119411 32314 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:20.119549 32314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:20.119841 32314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:20.122790 32314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "bf279e244bbb495e8123e393e9a1117c"
format_stamp: "Formatted at 2026-06-21 02:55:20 on dist-test-slave-bc52"
server_key: "0d7614b50da3e139faf2b636eb2325ef"
server_key_iv: "5c41b03dd15981d474df75d2d5ae2bb4"
server_key_version: "encryptionkey@0"
I20260621 02:55:20.123549 32314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "bf279e244bbb495e8123e393e9a1117c"
format_stamp: "Formatted at 2026-06-21 02:55:20 on dist-test-slave-bc52"
server_key: "0d7614b50da3e139faf2b636eb2325ef"
server_key_iv: "5c41b03dd15981d474df75d2d5ae2bb4"
server_key_version: "encryptionkey@0"
I20260621 02:55:20.128608 32314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:55:20.132025 32329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:20.133365 32314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:20.133540 32314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "bf279e244bbb495e8123e393e9a1117c"
format_stamp: "Formatted at 2026-06-21 02:55:20 on dist-test-slave-bc52"
server_key: "0d7614b50da3e139faf2b636eb2325ef"
server_key_iv: "5c41b03dd15981d474df75d2d5ae2bb4"
server_key_version: "encryptionkey@0"
I20260621 02:55:20.133737 32314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:20.139010 32314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:20.142760 32314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:20.143030 32314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:20.160485 32314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40631
I20260621 02:55:20.160527 32380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40631 every 8 connection(s)
I20260621 02:55:20.161864 32314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:20.164234 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32314
I20260621 02:55:20.164413 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:20.164958 11700 external_mini_cluster.cc:1428] Setting key 275c3e9f2789cb13d0d89c1cc1090fc5
I20260621 02:55:20.166182 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:20.173386 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:20.166711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33361 (local address 127.0.0.1:40631)
0621 02:55:20.167498 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:20.167513 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:20.167562 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:55:20.169570 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:20.169597 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:20.170046 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:20.170491 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:20.171439 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:20.171448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:20.171464 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:20.171480 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:20.171576 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:20.171602 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:20.171813 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:20.171947 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 02:55:20.172143 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":330,"threads_started":1}
I20260621 02:55:20.176757 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:20.180663 32381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c: Bootstrap starting.
I20260621 02:55:20.184432 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:20.181964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57957 (local address 127.0.0.1:40631)
0621 02:55:20.182128 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:20.182136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:20.182662 (+ 526us) server_negotiation.cc:407] Connection header received
0621 02:55:20.182827 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:20.182830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:20.182871 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:20.182950 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:20.183826 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:20.183830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:20.183833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:20.183846 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:20.183880 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:20.183894 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:20.183975 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:20.184110 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 02:55:20.184159 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260621 02:55:20.184620 32381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:20.185954 32381 log.cc:824] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:20.188344 32381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c: No bootstrap required, opened a new log
I20260621 02:55:20.195441 32381 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf279e244bbb495e8123e393e9a1117c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40631 } }
I20260621 02:55:20.195909 32381 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:20.196043 32381 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf279e244bbb495e8123e393e9a1117c, State: Initialized, Role: FOLLOWER
I20260621 02:55:20.196640 32381 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf279e244bbb495e8123e393e9a1117c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40631 } }
I20260621 02:55:20.196867 32381 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:20.197014 32381 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:20.197211 32381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:20.199239 32381 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf279e244bbb495e8123e393e9a1117c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40631 } }
I20260621 02:55:20.199703 32381 leader_election.cc:302] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf279e244bbb495e8123e393e9a1117c; no voters:
I20260621 02:55:20.200234 32381 leader_election.cc:288] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:20.200361 32391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:20.200990 32391 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [term 1 LEADER]: Becoming Leader. State: Replica: bf279e244bbb495e8123e393e9a1117c, State: Running, Role: LEADER
I20260621 02:55:20.201505 32391 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf279e244bbb495e8123e393e9a1117c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40631 } }
I20260621 02:55:20.202178 32381 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:20.204087 32393 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf279e244bbb495e8123e393e9a1117c. Latest consensus state: current_term: 1 leader_uuid: "bf279e244bbb495e8123e393e9a1117c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf279e244bbb495e8123e393e9a1117c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40631 } } }
I20260621 02:55:20.204283 32392 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf279e244bbb495e8123e393e9a1117c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf279e244bbb495e8123e393e9a1117c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40631 } } }
I20260621 02:55:20.204550 32392 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:20.204550 32393 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:20.206176 32397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:20.211319 32397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:20.220139 32397 catalog_manager.cc:1347] Generated new cluster ID: b4dc02eae48044ffb47984b1b60f7d36
I20260621 02:55:20.220242 32397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:20.240841 32397 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:20.241328 32397 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:20.248342 32397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf279e244bbb495e8123e393e9a1117c: Generated new TSK 0
I20260621 02:55:20.249274 32397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:20.258116 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32314
W20260621 02:55:20.259256 32376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33361
W20260621 02:55:20.259933 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:20.260075 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32314
2026-06-21T02:55:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1826 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:20 dist-test-slave-bc52 krb5kdc[32412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:20 dist-test-slave-bc52 krb5kdc[32412](info): set up 2 sockets
Jun 21 02:55:20 dist-test-slave-bc52 krb5kdc[32412](info): commencing operation
krb5kdc: starting...
W20260621 02:55:21.358371 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:21 dist-test-slave-bc52 krb5kdc[32412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:21.594600 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:21.918267 32425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:21.918757 32425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:21.918951 32425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:21.926353 32425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:21.926447 32425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:21.926553 32425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:21.926659 32425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:21.926750 32425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:21.936376 32425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:21.937845 32425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:21.939258 32425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:21.950932 32431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:21.951006 32430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:21.951006 32433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:21.952011 32425 server_base.cc:1034] running on GCE node
I20260621 02:55:21.952726 32425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:21.954109 32425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:21.955348 32425 hybrid_clock.cc:648] HybridClock initialized: now 1782010521955294 us; error 36 us; skew 500 ppm
I20260621 02:55:21.955687 32425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:21 dist-test-slave-bc52 krb5kdc[32412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:21.961767 32425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:21.963233 32425 webserver.cc:460] Webserver started at http://127.0.0.1:40687/ using document root <none> and password file <none>
I20260621 02:55:21.963905 32425 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:21.964048 32425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:21.964352 32425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:21.967280 32425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "2369fe73e77d41fba0be7c8a473ff629"
format_stamp: "Formatted at 2026-06-21 02:55:21 on dist-test-slave-bc52"
server_key: "3c617c1ff7389616c6b5cfa8363985f1"
server_key_iv: "1cbb2a388c0e9528006aa574c18a0975"
server_key_version: "encryptionkey@0"
I20260621 02:55:21.968036 32425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "2369fe73e77d41fba0be7c8a473ff629"
format_stamp: "Formatted at 2026-06-21 02:55:21 on dist-test-slave-bc52"
server_key: "3c617c1ff7389616c6b5cfa8363985f1"
server_key_iv: "1cbb2a388c0e9528006aa574c18a0975"
server_key_version: "encryptionkey@0"
I20260621 02:55:21.972987 32425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:55:21.976300 32440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:21.977593 32425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:21.977782 32425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "2369fe73e77d41fba0be7c8a473ff629"
format_stamp: "Formatted at 2026-06-21 02:55:21 on dist-test-slave-bc52"
server_key: "3c617c1ff7389616c6b5cfa8363985f1"
server_key_iv: "1cbb2a388c0e9528006aa574c18a0975"
server_key_version: "encryptionkey@0"
I20260621 02:55:21.977972 32425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:21.993012 32425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:21.997231 32425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:21.997505 32425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:22.015235 32425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33223
I20260621 02:55:22.015264 32491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33223 every 8 connection(s)
I20260621 02:55:22.016521 32425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:22.020008 32492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:22.026788 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32425
I20260621 02:55:22.026975 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:22.027474 11700 external_mini_cluster.cc:1428] Setting key 164b5635dd12bc3cec9fe5821c13afdb
I20260621 02:55:22.030325 32492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629: Bootstrap starting.
I20260621 02:55:22.034283 32492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:22.035328 32492 log.cc:824] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:22.037937 32492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629: No bootstrap required, opened a new log
Jun 21 02:55:22 dist-test-slave-bc52 krb5kdc[32412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:55:22.045307 32492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2369fe73e77d41fba0be7c8a473ff629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260621 02:55:22.045631 32492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:22.045737 32492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2369fe73e77d41fba0be7c8a473ff629, State: Initialized, Role: FOLLOWER
I20260621 02:55:22.046275 32492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2369fe73e77d41fba0be7c8a473ff629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260621 02:55:22.046576 32492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:22.046732 32492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:22.046903 32492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:22.049017 32492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2369fe73e77d41fba0be7c8a473ff629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260621 02:55:22.049517 32492 leader_election.cc:302] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2369fe73e77d41fba0be7c8a473ff629; no voters:
I20260621 02:55:22.050132 32492 leader_election.cc:288] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:22.050267 32497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:22.051493 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:22.029451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33437 (local address 127.0.0.1:33223)
0621 02:55:22.030387 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:22.030405 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:22.030771 (+ 366us) server_negotiation.cc:407] Connection header received
0621 02:55:22.033333 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:22.033362 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:22.033830 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:22.034231 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:22.035579 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:22.036893 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:22.038234 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:22.038892 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:22.039083 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:22.043226 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:22.043233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:22.043246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:22.043283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:22.046840 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:22.047566 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:22.047577 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:22.047588 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:22.047788 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:22.048187 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:22.048193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:22.048196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:22.048697 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:22.049013 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:22.049459 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 02:55:22.049882 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":417,"threads_started":1}
I20260621 02:55:22.052241 32492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:22.053555 32497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [term 1 LEADER]: Becoming Leader. State: Replica: 2369fe73e77d41fba0be7c8a473ff629, State: Running, Role: LEADER
I20260621 02:55:22.054087 32497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2369fe73e77d41fba0be7c8a473ff629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260621 02:55:22.056371 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:22.061736 32498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2369fe73e77d41fba0be7c8a473ff629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2369fe73e77d41fba0be7c8a473ff629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } } }
I20260621 02:55:22.062115 32498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:22.062685 32499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2369fe73e77d41fba0be7c8a473ff629. Latest consensus state: current_term: 1 leader_uuid: "2369fe73e77d41fba0be7c8a473ff629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2369fe73e77d41fba0be7c8a473ff629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } } }
I20260621 02:55:22.063802 32499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:22.067471 32507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:22.076498 32507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:22.093539 32507 catalog_manager.cc:1347] Generated new cluster ID: 24fe244cc96e4e888c0aa786e12f6a6f
I20260621 02:55:22.093667 32507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:22.107227 32507 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:22.109087 32507 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:22.113690 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:22.080946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53799 (local address 127.0.0.1:33223)
0621 02:55:22.081121 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:22.081131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:22.090563 (+ 9432us) server_negotiation.cc:407] Connection header received
0621 02:55:22.090831 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:22.090838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:22.090933 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:22.091053 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:22.093887 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:22.095419 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:22.098025 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:22.099000 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:22.099143 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:22.103545 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:22.103555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:22.103562 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:22.103617 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:22.108807 (+ 5190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:22.110337 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:22.110346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:22.110352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:22.110552 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:22.111175 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:22.111188 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:22.111194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:22.111703 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:22.112008 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:22.112766 (+ 758us) server_negotiation.cc:299] Negotiation successful
0621 02:55:22.113125 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:55:22.120421 32507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2369fe73e77d41fba0be7c8a473ff629: Generated new TSK 0
I20260621 02:55:22.121356 32507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:22.131275 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32425
I20260621 02:55:22.132262 32487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:22.221082 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32425
2026-06-21T02:55:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:22 dist-test-slave-bc52 krb5kdc[32525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:22 dist-test-slave-bc52 krb5kdc[32525](info): set up 2 sockets
Jun 21 02:55:22 dist-test-slave-bc52 krb5kdc[32525](info): commencing operation
krb5kdc: starting...
W20260621 02:55:23.297600 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:23 dist-test-slave-bc52 krb5kdc[32525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:23.532104 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:23.847566 32538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:23.847885 32538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:23.848040 32538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:23.855163 32538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:23.855306 32538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:23.855406 32538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:23.855530 32538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:23.855603 32538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:23.865154 32538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:23.866402 32538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:23.867657 32538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:23.879093 32543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:23.879135 32544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:23.879158 32546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:23.880159 32538 server_base.cc:1034] running on GCE node
I20260621 02:55:23.880927 32538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:23.882266 32538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:23.883491 32538 hybrid_clock.cc:648] HybridClock initialized: now 1782010523883458 us; error 44 us; skew 500 ppm
I20260621 02:55:23.883855 32538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:23 dist-test-slave-bc52 krb5kdc[32525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:23.886852 32538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:23.888203 32538 webserver.cc:460] Webserver started at http://127.0.0.1:44075/ using document root <none> and password file <none>
I20260621 02:55:23.888846 32538 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:23.888988 32538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:23.889343 32538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:23.892179 32538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c"
format_stamp: "Formatted at 2026-06-21 02:55:23 on dist-test-slave-bc52"
server_key: "111b569d3560b7cc372545846ece4ac6"
server_key_iv: "3260bd488448139c3e1001d8d2e73b7b"
server_key_version: "encryptionkey@0"
I20260621 02:55:23.892936 32538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c"
format_stamp: "Formatted at 2026-06-21 02:55:23 on dist-test-slave-bc52"
server_key: "111b569d3560b7cc372545846ece4ac6"
server_key_iv: "3260bd488448139c3e1001d8d2e73b7b"
server_key_version: "encryptionkey@0"
I20260621 02:55:23.897882 32538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:55:23.901046 32553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:23.902315 32538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:55:23.902487 32538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c"
format_stamp: "Formatted at 2026-06-21 02:55:23 on dist-test-slave-bc52"
server_key: "111b569d3560b7cc372545846ece4ac6"
server_key_iv: "3260bd488448139c3e1001d8d2e73b7b"
server_key_version: "encryptionkey@0"
I20260621 02:55:23.902681 32538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:23.921396 32538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:23.925083 32538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:23.925377 32538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:23.943081 32538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40723
I20260621 02:55:23.943116 32604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40723 every 8 connection(s)
I20260621 02:55:23.944453 32538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:23.945060 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32538
I20260621 02:55:23.945267 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:23.945860 11700 external_mini_cluster.cc:1428] Setting key 3b317cb71f4a9de61d0f6fae44e460ec
I20260621 02:55:23.948591 32605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:23.958660 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:23.947549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39201 (local address 127.0.0.1:40723)
0621 02:55:23.948288 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:23.948298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:23.948335 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:55:23.950085 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:23.950105 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:23.950456 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:23.950759 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:23.951319 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:23.952579 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:23.954442 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:23.955089 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:23.955215 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:23.956475 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:23.956483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:23.956495 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:23.956985 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:23.957060 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:23.957084 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:23.957235 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:23.957495 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 02:55:23.957740 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":332,"threads_started":1}
I20260621 02:55:23.960986 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c: Bootstrap starting.
I20260621 02:55:23.962303 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:23.964205 32605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:23.965387 32605 log.cc:824] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:23.968488 32605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c: No bootstrap required, opened a new log
I20260621 02:55:23.972936 32608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:23.966520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48823 (local address 127.0.0.1:40723)
0621 02:55:23.966727 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:23.966735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:23.967163 (+ 428us) server_negotiation.cc:407] Connection header received
0621 02:55:23.967323 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:23.967327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:23.967383 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:23.967454 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:23.968025 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:23.968774 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:23.970592 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:23.971051 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:23.971144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:23.972212 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:23.972216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:23.972223 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:23.972238 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:23.972273 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:23.972290 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:23.972361 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:23.972512 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 02:55:23.972579 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:55:23.976823 32605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260621 02:55:23.977165 32605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:23.977303 32605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017f36fe9a5e40a4a8dfbaf8a976a46c, State: Initialized, Role: FOLLOWER
I20260621 02:55:23.977902 32605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260621 02:55:23.978124 32605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:23.978278 32605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:23.978446 32605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:23.980791 32605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260621 02:55:23.981249 32605 leader_election.cc:302] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 017f36fe9a5e40a4a8dfbaf8a976a46c; no voters:
I20260621 02:55:23.981766 32605 leader_election.cc:288] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:23.981897 32615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:23.982609 32615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [term 1 LEADER]: Becoming Leader. State: Replica: 017f36fe9a5e40a4a8dfbaf8a976a46c, State: Running, Role: LEADER
I20260621 02:55:23.983174 32615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } }
I20260621 02:55:23.983773 32605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:23.986171 32617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 017f36fe9a5e40a4a8dfbaf8a976a46c. Latest consensus state: current_term: 1 leader_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } } }
I20260621 02:55:23.986621 32617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:23.987105 32616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017f36fe9a5e40a4a8dfbaf8a976a46c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40723 } } }
I20260621 02:55:23.987363 32616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:23.988438 32624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:23.992699 32624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:24.000419 32624 catalog_manager.cc:1347] Generated new cluster ID: 383fb6ed10ef4af78afdd49ea95f29c5
I20260621 02:55:24.000525 32624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:24.020219 32624 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:24.021978 32624 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:24.029232 32624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 017f36fe9a5e40a4a8dfbaf8a976a46c: Generated new TSK 0
I20260621 02:55:24.030211 32624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:24.040046 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32538
W20260621 02:55:24.041373 32600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39201
W20260621 02:55:24.042001 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:24.042147 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32538
2026-06-21T02:55:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:24 dist-test-slave-bc52 krb5kdc[32636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:24 dist-test-slave-bc52 krb5kdc[32636](info): set up 2 sockets
Jun 21 02:55:24 dist-test-slave-bc52 krb5kdc[32636](info): commencing operation
krb5kdc: starting...
W20260621 02:55:25.121579 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:25 dist-test-slave-bc52 krb5kdc[32636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:25.357900 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:25.676928 32649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:25.677212 32649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:25.677346 32649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:25.684108 32649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:25.684201 32649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:25.684244 32649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:25.684294 32649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:25.684336 32649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:25.693679 32649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:25.694844 32649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:25.696034 32649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:25.707261 32654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:25.707312 32655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:25.707346 32657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:25.708050 32649 server_base.cc:1034] running on GCE node
I20260621 02:55:25.708737 32649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:25.710093 32649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:25.711305 32649 hybrid_clock.cc:648] HybridClock initialized: now 1782010525711223 us; error 76 us; skew 500 ppm
I20260621 02:55:25.711596 32649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:25 dist-test-slave-bc52 krb5kdc[32636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:25.714387 32649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:25.715687 32649 webserver.cc:460] Webserver started at http://127.0.0.1:46341/ using document root <none> and password file <none>
I20260621 02:55:25.716280 32649 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:25.716410 32649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:25.716681 32649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:25.719456 32649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "61e6ef7127a7443a871ceadc9b48932c"
format_stamp: "Formatted at 2026-06-21 02:55:25 on dist-test-slave-bc52"
server_key: "9662c49e38ed95628c05c6869e769513"
server_key_iv: "047925991c152aecb82d6b8f4ee260db"
server_key_version: "encryptionkey@0"
I20260621 02:55:25.720189 32649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "61e6ef7127a7443a871ceadc9b48932c"
format_stamp: "Formatted at 2026-06-21 02:55:25 on dist-test-slave-bc52"
server_key: "9662c49e38ed95628c05c6869e769513"
server_key_iv: "047925991c152aecb82d6b8f4ee260db"
server_key_version: "encryptionkey@0"
I20260621 02:55:25.724736 32649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260621 02:55:25.727952 32664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:25.729133 32649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:55:25.729342 32649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "61e6ef7127a7443a871ceadc9b48932c"
format_stamp: "Formatted at 2026-06-21 02:55:25 on dist-test-slave-bc52"
server_key: "9662c49e38ed95628c05c6869e769513"
server_key_iv: "047925991c152aecb82d6b8f4ee260db"
server_key_version: "encryptionkey@0"
I20260621 02:55:25.729534 32649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:25.734740 32649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:25.738186 32649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:25.738461 32649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:25.755722 32649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43901
I20260621 02:55:25.755795 32715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43901 every 8 connection(s)
I20260621 02:55:25.757017 32649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:25.760321 32716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:25.761009 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32649
I20260621 02:55:25.761209 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:25.761613 11700 external_mini_cluster.cc:1428] Setting key bc48eeb412c7bf48a62fecacb45cbf39
I20260621 02:55:25.769920 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:25.763424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42085 (local address 127.0.0.1:43901)
0621 02:55:25.764402 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:25.764416 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:25.764751 (+ 335us) server_negotiation.cc:407] Connection header received
0621 02:55:25.766401 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:25.766428 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:25.766850 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:25.767273 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:25.768148 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:25.768155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:25.768170 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:25.768185 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:25.768243 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:25.768265 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:25.768455 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:25.768569 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 02:55:25.768749 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":477,"threads_started":1}
I20260621 02:55:25.773231 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:25.773518 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c: Bootstrap starting.
I20260621 02:55:25.777097 32716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:25.778376 32716 log.cc:824] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:25.781366 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:25.778399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58259 (local address 127.0.0.1:43901)
0621 02:55:25.778563 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:25.778568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:25.779113 (+ 545us) server_negotiation.cc:407] Connection header received
0621 02:55:25.779286 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:25.779292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:25.779350 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:25.779473 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:25.780575 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:25.780580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:25.780600 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:25.780619 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:25.780660 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:25.780695 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:25.780798 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:25.780963 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 02:55:25.781021 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:55:25.782028 32716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c: No bootstrap required, opened a new log
I20260621 02:55:25.791572 32716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e6ef7127a7443a871ceadc9b48932c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260621 02:55:25.791913 32716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:25.791985 32716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e6ef7127a7443a871ceadc9b48932c, State: Initialized, Role: FOLLOWER
I20260621 02:55:25.792457 32716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e6ef7127a7443a871ceadc9b48932c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260621 02:55:25.792649 32716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:25.792755 32716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:25.792871 32716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:25.794963 32716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e6ef7127a7443a871ceadc9b48932c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260621 02:55:25.795421 32716 leader_election.cc:302] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61e6ef7127a7443a871ceadc9b48932c; no voters:
I20260621 02:55:25.795943 32716 leader_election.cc:288] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:25.796089 32726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:25.797140 32726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [term 1 LEADER]: Becoming Leader. State: Replica: 61e6ef7127a7443a871ceadc9b48932c, State: Running, Role: LEADER
I20260621 02:55:25.797712 32726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e6ef7127a7443a871ceadc9b48932c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } }
I20260621 02:55:25.798074 32716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:25.800622 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61e6ef7127a7443a871ceadc9b48932c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e6ef7127a7443a871ceadc9b48932c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } } }
I20260621 02:55:25.800601 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61e6ef7127a7443a871ceadc9b48932c. Latest consensus state: current_term: 1 leader_uuid: "61e6ef7127a7443a871ceadc9b48932c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e6ef7127a7443a871ceadc9b48932c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43901 } } }
I20260621 02:55:25.801103 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:25.801108 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:25.802474 32731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:25.807974 32731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:25.816386 32731 catalog_manager.cc:1347] Generated new cluster ID: c1926662bace4b959d53833d0c7a0347
I20260621 02:55:25.816484 32731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:25.840767 32731 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:25.841250 32731 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:25.854609 32731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61e6ef7127a7443a871ceadc9b48932c: Generated new TSK 0
I20260621 02:55:25.855593 32731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:25.866335 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32649
W20260621 02:55:25.867424 32711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42085
W20260621 02:55:25.868091 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32649: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:25.868237 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32649
2026-06-21T02:55:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:25 dist-test-slave-bc52 krb5kdc[32747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:25 dist-test-slave-bc52 krb5kdc[32747](info): set up 2 sockets
Jun 21 02:55:25 dist-test-slave-bc52 krb5kdc[32747](info): commencing operation
krb5kdc: starting...
W20260621 02:55:26.965472 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:27 dist-test-slave-bc52 krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:27.200359 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:27.515589 32760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:27.515844 32760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:27.515993 32760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:27.522720 32760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:27.522828 32760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:27.522953 32760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:27.523047 32760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:27.523126 32760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:27.532359 32760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:27.533648 32760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:27.534914 32760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:27.546947 32766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:27.546968 32765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:27.546905 300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:27.547816 32760 server_base.cc:1034] running on GCE node
I20260621 02:55:27.548556 32760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:27.550016 32760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:27.551343 32760 hybrid_clock.cc:648] HybridClock initialized: now 1782010527551312 us; error 37 us; skew 500 ppm
I20260621 02:55:27.551703 32760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:27 dist-test-slave-bc52 krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:27.559149 32760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:27.560745 32760 webserver.cc:460] Webserver started at http://127.0.0.1:43613/ using document root <none> and password file <none>
I20260621 02:55:27.561458 32760 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:27.561594 32760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:27.561888 32760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:27.564703 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "00e65491db1240f9947d5e31fe50be17"
format_stamp: "Formatted at 2026-06-21 02:55:27 on dist-test-slave-bc52"
server_key: "7354e6255243fd5b65b98ac75c877768"
server_key_iv: "79b68c6ac33134629a74928d8780bf56"
server_key_version: "encryptionkey@0"
I20260621 02:55:27.565524 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "00e65491db1240f9947d5e31fe50be17"
format_stamp: "Formatted at 2026-06-21 02:55:27 on dist-test-slave-bc52"
server_key: "7354e6255243fd5b65b98ac75c877768"
server_key_iv: "79b68c6ac33134629a74928d8780bf56"
server_key_version: "encryptionkey@0"
I20260621 02:55:27.570427 32760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:55:27.573666 307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:27.574860 32760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:55:27.575031 32760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "00e65491db1240f9947d5e31fe50be17"
format_stamp: "Formatted at 2026-06-21 02:55:27 on dist-test-slave-bc52"
server_key: "7354e6255243fd5b65b98ac75c877768"
server_key_iv: "79b68c6ac33134629a74928d8780bf56"
server_key_version: "encryptionkey@0"
I20260621 02:55:27.575219 32760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:27.587689 32760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:27.591305 32760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:27.591553 32760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:27.608986 32760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37743
I20260621 02:55:27.609018 358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37743 every 8 connection(s)
I20260621 02:55:27.610327 32760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:27.610694 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 32760
I20260621 02:55:27.610872 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:27.611410 11700 external_mini_cluster.cc:1428] Setting key 597ecc0f7869d7714f93a0ed76ad5d42
I20260621 02:55:27.614612 359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:55:27 dist-test-slave-bc52 krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:55:27.628471 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17: Bootstrap starting.
I20260621 02:55:27.632915 359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:27.634416 359 log.cc:824] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:27.636600 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:27.613450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:38325 (local address 127.0.0.1:37743)
0621 02:55:27.614222 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:27.614234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:27.614274 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:55:27.615882 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:27.615903 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:27.616278 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:27.616583 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:27.618360 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:27.619673 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:27.621547 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:27.622196 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:27.622331 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:27.627387 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:27.627398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:27.627420 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:27.627476 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:27.631820 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:27.632611 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:27.632621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:27.632630 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:27.632869 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:27.633436 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:27.633442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:27.633445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:27.634190 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:27.634566 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:27.634910 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 02:55:27.635308 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":338,"threads_started":1}
I20260621 02:55:27.637723 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17: No bootstrap required, opened a new log
I20260621 02:55:27.640658 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:27.648701 359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e65491db1240f9947d5e31fe50be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 02:55:27.649225 359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:27.649339 359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e65491db1240f9947d5e31fe50be17, State: Initialized, Role: FOLLOWER
I20260621 02:55:27.649994 359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e65491db1240f9947d5e31fe50be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 02:55:27.650257 359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:27.650411 359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:27.650584 359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:27.653703 359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e65491db1240f9947d5e31fe50be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 02:55:27.654196 359 leader_election.cc:302] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e65491db1240f9947d5e31fe50be17; no voters:
I20260621 02:55:27.654974 359 leader_election.cc:288] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:27.655042 369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:27.656663 369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [term 1 LEADER]: Becoming Leader. State: Replica: 00e65491db1240f9947d5e31fe50be17, State: Running, Role: LEADER
I20260621 02:55:27.657330 369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e65491db1240f9947d5e31fe50be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260621 02:55:27.657639 359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:27.660266 371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e65491db1240f9947d5e31fe50be17. Latest consensus state: current_term: 1 leader_uuid: "00e65491db1240f9947d5e31fe50be17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e65491db1240f9947d5e31fe50be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } } }
I20260621 02:55:27.661522 371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:27.660977 370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e65491db1240f9947d5e31fe50be17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e65491db1240f9947d5e31fe50be17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } } }
I20260621 02:55:27.662417 370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:27.665052 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:27.646293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:60797 (local address 127.0.0.1:37743)
0621 02:55:27.646490 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:27.646502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:27.646867 (+ 365us) server_negotiation.cc:407] Connection header received
0621 02:55:27.646996 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:27.647001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:27.647088 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:27.647198 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:27.648476 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:27.649786 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:27.651185 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:27.651958 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:27.652105 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:27.654369 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:27.654376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:27.654381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:27.654416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:27.658796 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:27.660628 (+ 1832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:27.660633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:27.660637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:27.661164 (+ 527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:27.661836 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:27.661844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:27.661850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:27.662336 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:27.664087 (+ 1751us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:27.664303 (+ 216us) server_negotiation.cc:299] Negotiation successful
0621 02:55:27.664533 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:55:27.669632 378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:27.674800 378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:27.683763 378 catalog_manager.cc:1347] Generated new cluster ID: 1b74f013712b4a6599e09f0e0f0dac2a
I20260621 02:55:27.683882 378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:27.702970 378 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:27.704752 378 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:27.720323 378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e65491db1240f9947d5e31fe50be17: Generated new TSK 0
I20260621 02:55:27.721131 378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:27.735080 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 32760
I20260621 02:55:27.736351 354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:27.828701 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 32760
W20260621 02:55:27.845656 32745 connection.cc:535] client connection to 127.0.0.1:37743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37743): Connection reset by peer (error 104)
2026-06-21T02:55:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:27 dist-test-slave-bc52 krb5kdc[391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:27 dist-test-slave-bc52 krb5kdc[391](info): set up 2 sockets
Jun 21 02:55:27 dist-test-slave-bc52 krb5kdc[391](info): commencing operation
krb5kdc: starting...
W20260621 02:55:28.925619 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:29 dist-test-slave-bc52 krb5kdc[391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:29.157281 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:29.491638 404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:29.491901 404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:29.492098 404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:29.499044 404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:29.499159 404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:29.499289 404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:29.499390 404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:29.499461 404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:29.508798 404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:29.510017 404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:29.511303 404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:29.522854 410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:29.522907 412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:29.522928 409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:29.524271 404 server_base.cc:1034] running on GCE node
I20260621 02:55:29.525012 404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:29.526430 404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:29.527616 404 hybrid_clock.cc:648] HybridClock initialized: now 1782010529527563 us; error 51 us; skew 500 ppm
I20260621 02:55:29.527937 404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:29 dist-test-slave-bc52 krb5kdc[391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:29.531039 404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:29.532473 404 webserver.cc:460] Webserver started at http://127.0.0.1:46729/ using document root <none> and password file <none>
I20260621 02:55:29.533102 404 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:29.533280 404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:29.533598 404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:29.536562 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "e3e8bae637aa404e8f4986d241459c24"
format_stamp: "Formatted at 2026-06-21 02:55:29 on dist-test-slave-bc52"
server_key: "3eca58e12e3ae80c84cbadf8c5e839aa"
server_key_iv: "49d7fbd2534737f500e4b561a9c155b5"
server_key_version: "encryptionkey@0"
I20260621 02:55:29.537358 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "e3e8bae637aa404e8f4986d241459c24"
format_stamp: "Formatted at 2026-06-21 02:55:29 on dist-test-slave-bc52"
server_key: "3eca58e12e3ae80c84cbadf8c5e839aa"
server_key_iv: "49d7fbd2534737f500e4b561a9c155b5"
server_key_version: "encryptionkey@0"
I20260621 02:55:29.542070 404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:55:29.545248 419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:29.546504 404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:55:29.546684 404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "e3e8bae637aa404e8f4986d241459c24"
format_stamp: "Formatted at 2026-06-21 02:55:29 on dist-test-slave-bc52"
server_key: "3eca58e12e3ae80c84cbadf8c5e839aa"
server_key_iv: "49d7fbd2534737f500e4b561a9c155b5"
server_key_version: "encryptionkey@0"
I20260621 02:55:29.546854 404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:29.562723 404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:29.567152 404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:29.567384 404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:29.584434 404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38281
I20260621 02:55:29.584456 470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38281 every 8 connection(s)
I20260621 02:55:29.585841 404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:29.586371 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 404
I20260621 02:55:29.586565 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:29.587154 11700 external_mini_cluster.cc:1428] Setting key 14e072cb0410c226aee187d2efc21380
I20260621 02:55:29.590217 471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:29.599462 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:29.588985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:58041 (local address 127.0.0.1:38281)
0621 02:55:29.589743 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:29.589755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:29.589794 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:55:29.591409 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:29.591430 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:29.591812 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:29.592160 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:29.592704 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:29.594002 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:29.595290 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:29.595921 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:29.596099 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:29.597336 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:29.597342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:29.597357 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:29.597884 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:29.597956 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:29.597979 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:29.598162 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:29.598435 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 02:55:29.598608 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260621 02:55:29.600768 471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24: Bootstrap starting.
I20260621 02:55:29.603128 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:29.603992 471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:29.604933 471 log.cc:824] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:29.607661 471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24: No bootstrap required, opened a new log
I20260621 02:55:29.614235 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:29.607357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:33371 (local address 127.0.0.1:38281)
0621 02:55:29.607549 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:29.607558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:29.608048 (+ 490us) server_negotiation.cc:407] Connection header received
0621 02:55:29.608242 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:29.608246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:29.608304 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:29.608381 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:29.608968 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:29.609793 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:29.611696 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:29.612136 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:29.612225 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:29.613428 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:29.613433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:29.613436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:29.613453 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:29.613489 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:29.613504 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:29.613601 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:29.613870 (+ 269us) server_negotiation.cc:299] Negotiation successful
0621 02:55:29.613927 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:55:29.617012 471 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e8bae637aa404e8f4986d241459c24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38281 } }
I20260621 02:55:29.617403 471 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:29.617480 471 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e8bae637aa404e8f4986d241459c24, State: Initialized, Role: FOLLOWER
I20260621 02:55:29.618026 471 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e8bae637aa404e8f4986d241459c24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38281 } }
I20260621 02:55:29.618223 471 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:29.618322 471 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:29.618446 471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:29.620469 471 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e8bae637aa404e8f4986d241459c24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38281 } }
I20260621 02:55:29.620899 471 leader_election.cc:302] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3e8bae637aa404e8f4986d241459c24; no voters:
I20260621 02:55:29.621558 471 leader_election.cc:288] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:29.621709 481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:29.622351 481 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [term 1 LEADER]: Becoming Leader. State: Replica: e3e8bae637aa404e8f4986d241459c24, State: Running, Role: LEADER
I20260621 02:55:29.622872 481 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e8bae637aa404e8f4986d241459c24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38281 } }
I20260621 02:55:29.623787 471 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:29.625659 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3e8bae637aa404e8f4986d241459c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e8bae637aa404e8f4986d241459c24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38281 } } }
I20260621 02:55:29.626024 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:29.627697 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3e8bae637aa404e8f4986d241459c24. Latest consensus state: current_term: 1 leader_uuid: "e3e8bae637aa404e8f4986d241459c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3e8bae637aa404e8f4986d241459c24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38281 } } }
I20260621 02:55:29.628041 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:29.628459 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:29.635084 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:29.642997 486 catalog_manager.cc:1347] Generated new cluster ID: 42aa85614352477dbec2c89cee976237
I20260621 02:55:29.643098 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:29.682443 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:29.684193 486 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:29.693370 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3e8bae637aa404e8f4986d241459c24: Generated new TSK 0
I20260621 02:55:29.694197 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:29.713155 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 404
W20260621 02:55:29.714771 466 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:58041
W20260621 02:55:29.715472 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 404: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:29.715605 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 404
W20260621 02:55:29.729101 389 connection.cc:535] client connection to 127.0.0.1:38281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38281): Connection reset by peer (error 104)
2026-06-21T02:55:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:29 dist-test-slave-bc52 krb5kdc[502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:29 dist-test-slave-bc52 krb5kdc[502](info): set up 2 sockets
Jun 21 02:55:29 dist-test-slave-bc52 krb5kdc[502](info): commencing operation
krb5kdc: starting...
W20260621 02:55:30.793500 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:30 dist-test-slave-bc52 krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:31.024251 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:31.340018 515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:31.340303 515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:31.340471 515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:31.347589 515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:31.347693 515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:31.347739 515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:31.347824 515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:31.347903 515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:31.357453 515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:31.358662 515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:31.359933 515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:31.371261 521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:31.371299 523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:31.371806 520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:31.372074 515 server_base.cc:1034] running on GCE node
I20260621 02:55:31.372799 515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:31.374188 515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:31.375388 515 hybrid_clock.cc:648] HybridClock initialized: now 1782010531375338 us; error 46 us; skew 500 ppm
I20260621 02:55:31.375737 515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:31 dist-test-slave-bc52 krb5kdc[502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:31.378695 515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:31.379961 515 webserver.cc:460] Webserver started at http://127.0.0.1:33189/ using document root <none> and password file <none>
I20260621 02:55:31.380576 515 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:31.380708 515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:31.380981 515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:31.383756 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "49fdee85d4be407492aa9da6ec732267"
format_stamp: "Formatted at 2026-06-21 02:55:31 on dist-test-slave-bc52"
server_key: "97254c1a1a31f71f999f33f768c56697"
server_key_iv: "50fefe3580333fbe9f48f59f9adf6d80"
server_key_version: "encryptionkey@0"
I20260621 02:55:31.384502 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "49fdee85d4be407492aa9da6ec732267"
format_stamp: "Formatted at 2026-06-21 02:55:31 on dist-test-slave-bc52"
server_key: "97254c1a1a31f71f999f33f768c56697"
server_key_iv: "50fefe3580333fbe9f48f59f9adf6d80"
server_key_version: "encryptionkey@0"
I20260621 02:55:31.389465 515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:55:31.392690 530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:31.394275 515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:31.394482 515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "49fdee85d4be407492aa9da6ec732267"
format_stamp: "Formatted at 2026-06-21 02:55:31 on dist-test-slave-bc52"
server_key: "97254c1a1a31f71f999f33f768c56697"
server_key_iv: "50fefe3580333fbe9f48f59f9adf6d80"
server_key_version: "encryptionkey@0"
I20260621 02:55:31.394696 515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:31.400156 515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:31.404198 515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:31.404465 515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:31.420972 515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44977
I20260621 02:55:31.421013 581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44977 every 8 connection(s)
I20260621 02:55:31.422334 515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:31.426107 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 515
I20260621 02:55:31.426275 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:31.426472 582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:31.426733 11700 external_mini_cluster.cc:1428] Setting key bd0f6630301bdd35b3b519dd42ef4cbd
I20260621 02:55:31.438573 582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267: Bootstrap starting.
I20260621 02:55:31.440478 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:31.428692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:47271 (local address 127.0.0.1:44977)
0621 02:55:31.429698 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:31.429713 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:31.433778 (+ 4065us) server_negotiation.cc:407] Connection header received
0621 02:55:31.436686 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:31.436713 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:31.437133 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:31.437614 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:31.438572 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:31.438580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:31.438598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:31.438620 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:31.438691 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:31.438719 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:31.438972 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:31.439100 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 02:55:31.439281 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":436,"threads_started":1}
I20260621 02:55:31.442178 582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:31.443312 582 log.cc:824] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:31.443898 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:31.445895 582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267: No bootstrap required, opened a new log
I20260621 02:55:31.450945 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:31.448301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:34135 (local address 127.0.0.1:44977)
0621 02:55:31.448492 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:31.448498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:31.449024 (+ 526us) server_negotiation.cc:407] Connection header received
0621 02:55:31.449238 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:31.449242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:31.449295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:31.449381 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:31.450296 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:31.450299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:31.450302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:31.450316 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:31.450349 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:31.450362 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:31.450451 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:31.450591 (+ 140us) server_negotiation.cc:299] Negotiation successful
0621 02:55:31.450637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:55:31.454646 582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fdee85d4be407492aa9da6ec732267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260621 02:55:31.455010 582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:31.455114 582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fdee85d4be407492aa9da6ec732267, State: Initialized, Role: FOLLOWER
I20260621 02:55:31.455619 582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fdee85d4be407492aa9da6ec732267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260621 02:55:31.455839 582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:31.455972 582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:31.456130 582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:31.458797 582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fdee85d4be407492aa9da6ec732267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260621 02:55:31.459283 582 leader_election.cc:302] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49fdee85d4be407492aa9da6ec732267; no voters:
I20260621 02:55:31.459854 582 leader_election.cc:288] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:31.459996 592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:31.460661 592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [term 1 LEADER]: Becoming Leader. State: Replica: 49fdee85d4be407492aa9da6ec732267, State: Running, Role: LEADER
I20260621 02:55:31.461158 592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fdee85d4be407492aa9da6ec732267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260621 02:55:31.461951 582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:31.463989 594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49fdee85d4be407492aa9da6ec732267. Latest consensus state: current_term: 1 leader_uuid: "49fdee85d4be407492aa9da6ec732267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fdee85d4be407492aa9da6ec732267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } } }
I20260621 02:55:31.464422 594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:31.464874 593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49fdee85d4be407492aa9da6ec732267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fdee85d4be407492aa9da6ec732267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } } }
I20260621 02:55:31.465224 593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:31.465319 599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:31.471310 599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:31.479907 599 catalog_manager.cc:1347] Generated new cluster ID: d92f0afe490c432a9232be70ca41e43a
I20260621 02:55:31.480008 599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:31.498432 599 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:31.498901 599 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:31.514990 599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49fdee85d4be407492aa9da6ec732267: Generated new TSK 0
I20260621 02:55:31.515825 599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:31.530699 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 515
W20260621 02:55:31.531657 577 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:47271
W20260621 02:55:31.532310 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:31.532469 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 515
2026-06-21T02:55:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1819 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:31 dist-test-slave-bc52 krb5kdc[613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:31 dist-test-slave-bc52 krb5kdc[613](info): set up 2 sockets
Jun 21 02:55:31 dist-test-slave-bc52 krb5kdc[613](info): commencing operation
krb5kdc: starting...
W20260621 02:55:32.637462 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:32 dist-test-slave-bc52 krb5kdc[613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:32.871836 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:33.191326 626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:33.191637 626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:33.191797 626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:33.198820 626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:33.198964 626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:33.199057 626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:33.199154 626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:33.199223 626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:33.208547 626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:33.209852 626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:33.211086 626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:33.222671 631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:33.222625 632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:33.222704 634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:33.223731 626 server_base.cc:1034] running on GCE node
I20260621 02:55:33.224452 626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:33.225863 626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:33.227082 626 hybrid_clock.cc:648] HybridClock initialized: now 1782010533227045 us; error 42 us; skew 500 ppm
I20260621 02:55:33.227416 626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:33 dist-test-slave-bc52 krb5kdc[613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:33.233747 626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:33.235025 626 webserver.cc:460] Webserver started at http://127.0.0.1:39695/ using document root <none> and password file <none>
I20260621 02:55:33.235611 626 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:33.235705 626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:33.235935 626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:33.238746 626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "cc2ff703a56f446ea9467ad557d3bff0"
format_stamp: "Formatted at 2026-06-21 02:55:33 on dist-test-slave-bc52"
server_key: "bc212ae6460cdd7a647d988c8d7ef513"
server_key_iv: "cac95e9f9858f6976de051471a1def42"
server_key_version: "encryptionkey@0"
I20260621 02:55:33.239439 626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "cc2ff703a56f446ea9467ad557d3bff0"
format_stamp: "Formatted at 2026-06-21 02:55:33 on dist-test-slave-bc52"
server_key: "bc212ae6460cdd7a647d988c8d7ef513"
server_key_iv: "cac95e9f9858f6976de051471a1def42"
server_key_version: "encryptionkey@0"
I20260621 02:55:33.244194 626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 02:55:33.247447 641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:33.248625 626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260621 02:55:33.248793 626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "cc2ff703a56f446ea9467ad557d3bff0"
format_stamp: "Formatted at 2026-06-21 02:55:33 on dist-test-slave-bc52"
server_key: "bc212ae6460cdd7a647d988c8d7ef513"
server_key_iv: "cac95e9f9858f6976de051471a1def42"
server_key_version: "encryptionkey@0"
I20260621 02:55:33.248975 626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:33.269869 626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:33.274052 626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:33.274267 626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:33.291926 626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44781
I20260621 02:55:33.291965 692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44781 every 8 connection(s)
I20260621 02:55:33.293165 626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:33.294200 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 626
I20260621 02:55:33.294387 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:33.294824 11700 external_mini_cluster.cc:1428] Setting key 960b00cc6c26f7504e57b2a6a754df39
I20260621 02:55:33.297539 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:55:33 dist-test-slave-bc52 krb5kdc[613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:55:33.310467 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0: Bootstrap starting.
I20260621 02:55:33.314711 693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:33.315899 693 log.cc:824] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:33.319067 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:33.296563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:55365 (local address 127.0.0.1:44781)
0621 02:55:33.297328 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:33.297339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:33.297378 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:55:33.298980 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:33.299001 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:33.299359 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:33.299666 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:33.301408 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:33.302715 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:33.304487 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:33.305136 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:33.305353 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:33.310302 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:33.310322 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:33.310341 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:33.310390 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:33.314250 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:33.315229 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:33.315238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:33.315245 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:33.315449 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:33.315846 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:33.315851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:33.315854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:33.316635 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:33.316859 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:33.317532 (+ 673us) server_negotiation.cc:299] Negotiation successful
0621 02:55:33.317839 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":343,"threads_started":1}
I20260621 02:55:33.319203 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0: No bootstrap required, opened a new log
I20260621 02:55:33.322788 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:33.327162 693 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2ff703a56f446ea9467ad557d3bff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260621 02:55:33.327598 693 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:33.327721 693 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2ff703a56f446ea9467ad557d3bff0, State: Initialized, Role: FOLLOWER
I20260621 02:55:33.328357 693 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2ff703a56f446ea9467ad557d3bff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260621 02:55:33.328635 693 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:33.328769 693 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:33.328965 693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:33.332221 693 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2ff703a56f446ea9467ad557d3bff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260621 02:55:33.332774 693 leader_election.cc:302] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2ff703a56f446ea9467ad557d3bff0; no voters:
I20260621 02:55:33.333455 693 leader_election.cc:288] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:33.333528 703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:33.335093 703 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [term 1 LEADER]: Becoming Leader. State: Replica: cc2ff703a56f446ea9467ad557d3bff0, State: Running, Role: LEADER
I20260621 02:55:33.335582 703 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2ff703a56f446ea9467ad557d3bff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } }
I20260621 02:55:33.335759 693 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:33.338517 704 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2ff703a56f446ea9467ad557d3bff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2ff703a56f446ea9467ad557d3bff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } } }
I20260621 02:55:33.338908 704 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:33.342664 705 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2ff703a56f446ea9467ad557d3bff0. Latest consensus state: current_term: 1 leader_uuid: "cc2ff703a56f446ea9467ad557d3bff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2ff703a56f446ea9467ad557d3bff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44781 } } }
I20260621 02:55:33.342970 705 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:33.344498 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:33.326828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:49621 (local address 127.0.0.1:44781)
0621 02:55:33.327008 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:33.327014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:33.327553 (+ 539us) server_negotiation.cc:407] Connection header received
0621 02:55:33.327742 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:33.327748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:33.327842 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:33.327931 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:33.329726 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:33.330502 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:33.332310 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:33.332801 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:33.332891 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:33.336131 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:33.336137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:33.336142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:33.336176 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:33.340346 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:33.341589 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:33.341597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:33.341603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:33.341779 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:33.342509 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:33.342521 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:33.342526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:33.342950 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:33.343322 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:33.343864 (+ 542us) server_negotiation.cc:299] Negotiation successful
0621 02:55:33.344230 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:55:33.344568 709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:33.349550 709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:33.358940 709 catalog_manager.cc:1347] Generated new cluster ID: 4180911a98a0491bba2c2598665562de
I20260621 02:55:33.359050 709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:33.374109 709 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:33.375288 709 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:33.390095 709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2ff703a56f446ea9467ad557d3bff0: Generated new TSK 0
I20260621 02:55:33.391081 709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:33.404115 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 626
I20260621 02:55:33.405084 688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:33.494087 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 626
W20260621 02:55:33.507982 611 connection.cc:535] client connection to 127.0.0.1:44781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44781): Connection reset by peer (error 104)
2026-06-21T02:55:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:33 dist-test-slave-bc52 krb5kdc[725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:33 dist-test-slave-bc52 krb5kdc[725](info): set up 2 sockets
Jun 21 02:55:33 dist-test-slave-bc52 krb5kdc[725](info): commencing operation
krb5kdc: starting...
W20260621 02:55:34.573578 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:34 dist-test-slave-bc52 krb5kdc[725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:34.803503 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:35.128835 738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:35.129137 738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:35.129285 738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:35.136349 738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:35.136440 738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:35.136483 738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:35.136533 738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:35.136567 738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:35.145856 738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:35.146948 738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:35.148159 738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:35.159559 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:35.159559 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:35.159595 743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:35.160341 738 server_base.cc:1034] running on GCE node
I20260621 02:55:35.160989 738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:35.162292 738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:35.163496 738 hybrid_clock.cc:648] HybridClock initialized: now 1782010535163451 us; error 36 us; skew 500 ppm
I20260621 02:55:35.163828 738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:35 dist-test-slave-bc52 krb5kdc[725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:35.167060 738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:35.168397 738 webserver.cc:460] Webserver started at http://127.0.0.1:37949/ using document root <none> and password file <none>
I20260621 02:55:35.169039 738 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:35.169230 738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:35.169538 738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:35.172364 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "24b180c909be46e78fa16ae75236951e"
format_stamp: "Formatted at 2026-06-21 02:55:35 on dist-test-slave-bc52"
server_key: "ed9735ab5993eba410a04ea162f6f228"
server_key_iv: "3e4f156949dfc87047b229033cbfd7f2"
server_key_version: "encryptionkey@0"
I20260621 02:55:35.173098 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "24b180c909be46e78fa16ae75236951e"
format_stamp: "Formatted at 2026-06-21 02:55:35 on dist-test-slave-bc52"
server_key: "ed9735ab5993eba410a04ea162f6f228"
server_key_iv: "3e4f156949dfc87047b229033cbfd7f2"
server_key_version: "encryptionkey@0"
I20260621 02:55:35.177858 738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:55:35.180968 753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:35.182329 738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:55:35.182498 738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "24b180c909be46e78fa16ae75236951e"
format_stamp: "Formatted at 2026-06-21 02:55:35 on dist-test-slave-bc52"
server_key: "ed9735ab5993eba410a04ea162f6f228"
server_key_iv: "3e4f156949dfc87047b229033cbfd7f2"
server_key_version: "encryptionkey@0"
I20260621 02:55:35.182686 738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:35.199050 738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:35.203271 738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:35.203557 738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:35.221302 738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40887
I20260621 02:55:35.221342 804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40887 every 8 connection(s)
I20260621 02:55:35.222572 738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:35.223863 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 738
I20260621 02:55:35.224045 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:35.224573 11700 external_mini_cluster.cc:1428] Setting key c7bd1f8173b9c18e3a8a648b48dcd802
I20260621 02:55:35.226966 805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:35.237494 805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e: Bootstrap starting.
I20260621 02:55:35.237699 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:35.226359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:44721 (local address 127.0.0.1:40887)
0621 02:55:35.227199 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:35.227210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:35.227249 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:55:35.229012 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:35.229038 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:35.229640 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:35.230093 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:35.230599 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:35.232063 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:35.233433 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:35.234070 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:35.234270 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:35.235483 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:35.235490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:35.235506 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:35.236013 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:35.236077 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:35.236099 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:35.236257 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:35.236570 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 02:55:35.236719 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":307,"threads_started":1}
I20260621 02:55:35.240562 805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:35.241288 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:35.241717 805 log.cc:824] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:35.244124 805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e: No bootstrap required, opened a new log
I20260621 02:55:35.252300 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:35.245666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:49303 (local address 127.0.0.1:40887)
0621 02:55:35.245887 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:35.245892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:35.246235 (+ 343us) server_negotiation.cc:407] Connection header received
0621 02:55:35.246447 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:35.246450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:35.246508 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:35.246584 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:35.247143 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:35.247875 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:35.249745 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:35.250192 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:35.250285 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:35.251477 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:35.251482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:35.251485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:35.251500 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:35.251549 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:35.251567 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:35.251651 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:35.251880 (+ 229us) server_negotiation.cc:299] Negotiation successful
0621 02:55:35.251945 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:55:35.254086 805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b180c909be46e78fa16ae75236951e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260621 02:55:35.254451 805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:35.254552 805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b180c909be46e78fa16ae75236951e, State: Initialized, Role: FOLLOWER
I20260621 02:55:35.255050 805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b180c909be46e78fa16ae75236951e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260621 02:55:35.255306 805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:35.255438 805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:35.255617 805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:35.257848 805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b180c909be46e78fa16ae75236951e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260621 02:55:35.258395 805 leader_election.cc:302] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b180c909be46e78fa16ae75236951e; no voters:
I20260621 02:55:35.259043 805 leader_election.cc:288] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:35.259183 815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:35.259920 815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [term 1 LEADER]: Becoming Leader. State: Replica: 24b180c909be46e78fa16ae75236951e, State: Running, Role: LEADER
I20260621 02:55:35.260416 815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b180c909be46e78fa16ae75236951e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } }
I20260621 02:55:35.261086 805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:35.263376 817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b180c909be46e78fa16ae75236951e. Latest consensus state: current_term: 1 leader_uuid: "24b180c909be46e78fa16ae75236951e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b180c909be46e78fa16ae75236951e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } } }
I20260621 02:55:35.263738 817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:35.263513 816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b180c909be46e78fa16ae75236951e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b180c909be46e78fa16ae75236951e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40887 } } }
I20260621 02:55:35.263924 816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e [sys.catalog]: This master's current role is: LEADER
W20260621 02:55:35.272485 831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:55:35.272619 831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:55:35.272759 823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:35.274513 823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:35.281946 823 catalog_manager.cc:1347] Generated new cluster ID: 303f08943e934f9cbd0cecd2b07ded63
I20260621 02:55:35.282037 823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:35.323009 823 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:35.324186 823 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:35.334164 823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b180c909be46e78fa16ae75236951e: Generated new TSK 0
I20260621 02:55:35.334923 823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:35.346508 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 738
W20260621 02:55:35.347663 800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:44721
W20260621 02:55:35.348389 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 738: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:35.348531 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 738
W20260621 02:55:35.365420 723 connection.cc:535] client connection to 127.0.0.1:40887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40887): Connection reset by peer (error 104)
2026-06-21T02:55:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:35 dist-test-slave-bc52 krb5kdc[836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:35 dist-test-slave-bc52 krb5kdc[836](info): set up 2 sockets
Jun 21 02:55:35 dist-test-slave-bc52 krb5kdc[836](info): commencing operation
krb5kdc: starting...
W20260621 02:55:36.449591 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:36 dist-test-slave-bc52 krb5kdc[836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:55:36.690322 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:37.008241 849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:37.008560 849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:37.008734 849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:37.015810 849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:37.015954 849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:37.016057 849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:37.016160 849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:37.016237 849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:37.025885 849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:37.027132 849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:37.028398 849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:37.040162 854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:37.040226 855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:37.040800 849 server_base.cc:1034] running on GCE node
W20260621 02:55:37.040894 857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:37.041774 849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:37.043262 849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:37.044505 849 hybrid_clock.cc:648] HybridClock initialized: now 1782010537044453 us; error 42 us; skew 500 ppm
I20260621 02:55:37.044855 849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:37 dist-test-slave-bc52 krb5kdc[836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:37.047891 849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:55:37.049365 849 webserver.cc:460] Webserver started at http://127.0.0.1:36443/ using document root <none> and password file <none>
I20260621 02:55:37.050031 849 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:37.050176 849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:37.050464 849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:37.053366 849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "c96ac1e0fb9143248744decc2885214e"
format_stamp: "Formatted at 2026-06-21 02:55:37 on dist-test-slave-bc52"
server_key: "275abce802137809b8a6c81da120fe33"
server_key_iv: "aa2fc82699beb443bddf351eea6007bf"
server_key_version: "encryptionkey@0"
I20260621 02:55:37.054116 849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "c96ac1e0fb9143248744decc2885214e"
format_stamp: "Formatted at 2026-06-21 02:55:37 on dist-test-slave-bc52"
server_key: "275abce802137809b8a6c81da120fe33"
server_key_iv: "aa2fc82699beb443bddf351eea6007bf"
server_key_version: "encryptionkey@0"
I20260621 02:55:37.058957 849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 02:55:37.062213 864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:37.063570 849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:55:37.063755 849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "c96ac1e0fb9143248744decc2885214e"
format_stamp: "Formatted at 2026-06-21 02:55:37 on dist-test-slave-bc52"
server_key: "275abce802137809b8a6c81da120fe33"
server_key_iv: "aa2fc82699beb443bddf351eea6007bf"
server_key_version: "encryptionkey@0"
I20260621 02:55:37.063953 849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:37.069381 849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:37.073055 849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:37.073403 849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:37.091611 849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39293
I20260621 02:55:37.091657 915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39293 every 8 connection(s)
I20260621 02:55:37.092998 849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:37.096624 916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:37.103396 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 849
I20260621 02:55:37.103591 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:37.104135 11700 external_mini_cluster.cc:1428] Setting key 0d7096c228395223928ce2378b0ad419
I20260621 02:55:37.108067 916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e: Bootstrap starting.
I20260621 02:55:37.111464 916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:37.112054 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:37.106126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:33795 (local address 127.0.0.1:39293)
0621 02:55:37.106897 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:37.106909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:37.106952 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:55:37.108684 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:37.108706 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:37.109051 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:37.109485 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:37.110504 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:37.110510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:37.110522 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:37.110535 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:37.110590 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:37.110608 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:37.110796 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:37.110930 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 02:55:37.111113 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":323,"threads_started":1}
I20260621 02:55:37.112591 916 log.cc:824] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:37.115582 916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e: No bootstrap required, opened a new log
I20260621 02:55:37.115587 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:37.122280 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:37.119879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:40633 (local address 127.0.0.1:39293)
0621 02:55:37.120085 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:37.120091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:37.120411 (+ 320us) server_negotiation.cc:407] Connection header received
0621 02:55:37.120572 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:37.120575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:37.120617 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:37.120695 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:37.121649 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:37.121653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:37.121656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:37.121669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:37.121703 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:37.121716 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:37.121807 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:37.121922 (+ 115us) server_negotiation.cc:299] Negotiation successful
0621 02:55:37.121969 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:55:37.124135 916 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96ac1e0fb9143248744decc2885214e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260621 02:55:37.124495 916 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:37.124599 916 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96ac1e0fb9143248744decc2885214e, State: Initialized, Role: FOLLOWER
I20260621 02:55:37.125133 916 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96ac1e0fb9143248744decc2885214e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260621 02:55:37.125437 916 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:37.125537 916 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:37.125660 916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:37.127993 916 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96ac1e0fb9143248744decc2885214e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260621 02:55:37.128412 916 leader_election.cc:302] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96ac1e0fb9143248744decc2885214e; no voters:
I20260621 02:55:37.128877 916 leader_election.cc:288] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:37.129004 926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:37.129669 926 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [term 1 LEADER]: Becoming Leader. State: Replica: c96ac1e0fb9143248744decc2885214e, State: Running, Role: LEADER
I20260621 02:55:37.130196 926 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96ac1e0fb9143248744decc2885214e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260621 02:55:37.130888 916 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:37.133466 927 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96ac1e0fb9143248744decc2885214e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96ac1e0fb9143248744decc2885214e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } } }
I20260621 02:55:37.133941 927 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:37.134642 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96ac1e0fb9143248744decc2885214e. Latest consensus state: current_term: 1 leader_uuid: "c96ac1e0fb9143248744decc2885214e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96ac1e0fb9143248744decc2885214e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } } }
I20260621 02:55:37.135051 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:37.135430 935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:37.140501 935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:37.149816 935 catalog_manager.cc:1347] Generated new cluster ID: 5bc0a66450714fef9136f4fb02e302da
I20260621 02:55:37.149940 935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:37.160547 935 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:37.161062 935 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:37.172665 935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96ac1e0fb9143248744decc2885214e: Generated new TSK 0
I20260621 02:55:37.173640 935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:37.185913 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 849
W20260621 02:55:37.187281 911 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.147:33795
W20260621 02:55:37.187943 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 849: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:37.188078 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 849
2026-06-21T02:55:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1838 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:37 dist-test-slave-bc52 krb5kdc[947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:37 dist-test-slave-bc52 krb5kdc[947](info): set up 2 sockets
Jun 21 02:55:37 dist-test-slave-bc52 krb5kdc[947](info): commencing operation
krb5kdc: starting...
W20260621 02:55:38.269724 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:38 dist-test-slave-bc52 krb5kdc[947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:38.501060 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46773
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:46773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:38.815968 960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:38.816246 960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:38.816373 960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:38.823207 960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:38.823335 960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:38.823415 960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:38.823503 960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:38.823580 960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:38.832655 960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:46773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:46773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:38.834051 960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:38.835419 960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:38.847044 965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:38.847127 966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:38.847090 968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:38.848213 960 server_base.cc:1034] running on GCE node
I20260621 02:55:38.849061 960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:38.850421 960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:38.851624 960 hybrid_clock.cc:648] HybridClock initialized: now 1782010538851577 us; error 39 us; skew 500 ppm
I20260621 02:55:38.851974 960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:38 dist-test-slave-bc52 krb5kdc[947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010538, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:38.858019 960 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:38.859462 960 webserver.cc:460] Webserver started at http://127.11.109.62:40581/ using document root <none> and password file <none>
I20260621 02:55:38.860134 960 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:38.860280 960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:38.860569 960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:38.863477 960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "4686bd52c9594236aecccd84aebce8b7"
format_stamp: "Formatted at 2026-06-21 02:55:38 on dist-test-slave-bc52"
server_key: "b567193c5d991d573591ef1bbfcd6e09"
server_key_iv: "de15335e59339d9ca26fd21cef6631df"
server_key_version: "encryptionkey@0"
I20260621 02:55:38.864226 960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "4686bd52c9594236aecccd84aebce8b7"
format_stamp: "Formatted at 2026-06-21 02:55:38 on dist-test-slave-bc52"
server_key: "b567193c5d991d573591ef1bbfcd6e09"
server_key_iv: "de15335e59339d9ca26fd21cef6631df"
server_key_version: "encryptionkey@0"
I20260621 02:55:38.869084 960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 02:55:38.872505 975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:38.873786 960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:38.873965 960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "4686bd52c9594236aecccd84aebce8b7"
format_stamp: "Formatted at 2026-06-21 02:55:38 on dist-test-slave-bc52"
server_key: "b567193c5d991d573591ef1bbfcd6e09"
server_key_iv: "de15335e59339d9ca26fd21cef6631df"
server_key_version: "encryptionkey@0"
I20260621 02:55:38.874153 960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:38.889617 960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:38.893328 960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:38.893604 960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:38.911453 960 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:46773
I20260621 02:55:38.911475 1026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:46773 every 8 connection(s)
I20260621 02:55:38.912717 960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:38.916124 1027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:38.922504 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 960
I20260621 02:55:38.922683 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:38.923205 11700 external_mini_cluster.cc:1428] Setting key 9f4d331677b3377d1fbbc53195e74423
I20260621 02:55:38.926621 1027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7: Bootstrap starting.
I20260621 02:55:38.929476 1027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:38.930786 1027 log.cc:824] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:38.933475 1027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7: No bootstrap required, opened a new log
Jun 21 02:55:38 dist-test-slave-bc52 krb5kdc[947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:38.941422 1027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4686bd52c9594236aecccd84aebce8b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46773 } }
I20260621 02:55:38.941802 1027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:38.941905 1027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4686bd52c9594236aecccd84aebce8b7, State: Initialized, Role: FOLLOWER
I20260621 02:55:38.942512 1027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4686bd52c9594236aecccd84aebce8b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46773 } }
I20260621 02:55:38.942775 1027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:38.942931 1027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:38.943120 1027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:38.945773 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:38.924982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35057 (local address 127.11.109.62:46773)
0621 02:55:38.925733 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:38.925745 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:38.925783 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:55:38.927514 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:38.927535 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:38.927911 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:38.928285 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:38.929732 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:38.931666 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:38.932942 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:38.933972 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:38.934162 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:38.938409 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:38.938416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:38.938429 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:38.938464 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:38.941525 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:38.942418 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:38.942423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:38.942429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:38.942589 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:38.943106 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:38.943111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:38.943114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:38.943672 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:38.943911 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:38.944400 (+ 489us) server_negotiation.cc:299] Negotiation successful
0621 02:55:38.944677 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":367,"threads_started":1}
I20260621 02:55:38.945751 1027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4686bd52c9594236aecccd84aebce8b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46773 } }
I20260621 02:55:38.946213 1027 leader_election.cc:302] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4686bd52c9594236aecccd84aebce8b7; no voters:
I20260621 02:55:38.946787 1027 leader_election.cc:288] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:38.946923 1032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:38.948067 1032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4686bd52c9594236aecccd84aebce8b7, State: Running, Role: LEADER
I20260621 02:55:38.948624 1032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4686bd52c9594236aecccd84aebce8b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46773 } }
I20260621 02:55:38.949038 1027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:38.949945 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:38.955832 1034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4686bd52c9594236aecccd84aebce8b7. Latest consensus state: current_term: 1 leader_uuid: "4686bd52c9594236aecccd84aebce8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4686bd52c9594236aecccd84aebce8b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46773 } } }
I20260621 02:55:38.956205 1034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:38.955929 1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4686bd52c9594236aecccd84aebce8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4686bd52c9594236aecccd84aebce8b7" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 46773 } } }
I20260621 02:55:38.956336 1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:38.966085 1049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:38.971354 1049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:38.980315 1049 catalog_manager.cc:1347] Generated new cluster ID: 54c4a348d44a4a1faaa983ec8ecc1a8f
I20260621 02:55:38.980423 1049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:38.986240 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:38.967815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60219 (local address 127.11.109.62:46773)
0621 02:55:38.968035 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:38.968048 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:38.968581 (+ 533us) server_negotiation.cc:407] Connection header received
0621 02:55:38.968747 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:38.968752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:38.968822 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:38.968929 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:38.970829 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:38.972034 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:38.973800 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:38.974515 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:38.974635 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:38.979529 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:38.979534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:38.979537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:38.979566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:38.983215 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:38.984094 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:38.984103 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:38.984106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:38.984233 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:38.984699 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:38.984703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:38.984707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:38.985086 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:38.985369 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:38.985755 (+ 386us) server_negotiation.cc:299] Negotiation successful
0621 02:55:38.985924 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:55:39.000845 1049 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:39.002125 1049 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:39.012151 1049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4686bd52c9594236aecccd84aebce8b7: Generated new TSK 0
I20260621 02:55:39.013188 1049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:39.028116 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 960
I20260621 02:55:39.029495 1022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:39.120466 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 960
W20260621 02:55:39.134377 945 connection.cc:535] client connection to 127.11.109.62:46773 recv error: Network error: failed to read from TLS socket (remote: 127.11.109.62:46773): Connection reset by peer (error 104)
2026-06-21T02:55:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:39 dist-test-slave-bc52 krb5kdc[1059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:39 dist-test-slave-bc52 krb5kdc[1059](info): set up 2 sockets
Jun 21 02:55:39 dist-test-slave-bc52 krb5kdc[1059](info): commencing operation
krb5kdc: starting...
W20260621 02:55:40.229735 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:40 dist-test-slave-bc52 krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:40.472232 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41669
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:40.790920 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:40.791241 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:40.791407 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:40.798502 1072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:40.798651 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:40.798749 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:40.798853 1072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:40.798936 1072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:40.808349 1072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:41669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:40.809666 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:40.810931 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:40.822453 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:40.822412 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:40.822490 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:40.823199 1072 server_base.cc:1034] running on GCE node
I20260621 02:55:40.823935 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:40.825376 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:40.826604 1072 hybrid_clock.cc:648] HybridClock initialized: now 1782010540826551 us; error 46 us; skew 500 ppm
I20260621 02:55:40.826967 1072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:40 dist-test-slave-bc52 krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010540, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:40.830044 1072 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:40.831487 1072 webserver.cc:460] Webserver started at http://127.11.109.62:38741/ using document root <none> and password file <none>
I20260621 02:55:40.832161 1072 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:40.832311 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:40.832610 1072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:40.835565 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "6eabcd8cc5434a85a3be49cd14081a80"
format_stamp: "Formatted at 2026-06-21 02:55:40 on dist-test-slave-bc52"
server_key: "cda29d14080c3088c836169170eacf17"
server_key_iv: "b3d151d9926a3e1dbe22c73726e1e878"
server_key_version: "encryptionkey@0"
I20260621 02:55:40.836337 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "6eabcd8cc5434a85a3be49cd14081a80"
format_stamp: "Formatted at 2026-06-21 02:55:40 on dist-test-slave-bc52"
server_key: "cda29d14080c3088c836169170eacf17"
server_key_iv: "b3d151d9926a3e1dbe22c73726e1e878"
server_key_version: "encryptionkey@0"
I20260621 02:55:40.841519 1072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 02:55:40.844944 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:40.846326 1072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:40.846529 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "6eabcd8cc5434a85a3be49cd14081a80"
format_stamp: "Formatted at 2026-06-21 02:55:40 on dist-test-slave-bc52"
server_key: "cda29d14080c3088c836169170eacf17"
server_key_iv: "b3d151d9926a3e1dbe22c73726e1e878"
server_key_version: "encryptionkey@0"
I20260621 02:55:40.846724 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:40.865397 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:40.869601 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:40.869861 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:40.890981 1072 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:41669
I20260621 02:55:40.890988 1138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:41669 every 8 connection(s)
I20260621 02:55:40.892329 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:40.895689 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1072
I20260621 02:55:40.895911 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:40.896206 1139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:40.896387 11700 external_mini_cluster.cc:1428] Setting key e788b73e22261aa2e21c3cbb5ac0e53d
I20260621 02:55:40.911142 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:40.898162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38891 (local address 127.11.109.62:41669)
0621 02:55:40.898958 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:40.898973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:40.899028 (+ 55us) server_negotiation.cc:407] Connection header received
0621 02:55:40.901086 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:40.901115 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:40.901666 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:40.902093 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:40.902598 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:40.904610 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:40.906041 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:40.907087 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:40.907273 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:40.908417 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:40.908429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:40.908452 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:40.909209 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:40.909308 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:40.909336 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:40.909483 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:40.909714 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 02:55:40.909887 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260621 02:55:40.911303 1139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80: Bootstrap starting.
I20260621 02:55:40.914968 1139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:40.915093 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:40.916181 1139 log.cc:824] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:40.919443 1139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80: No bootstrap required, opened a new log
I20260621 02:55:40.926805 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:40.920125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37351 (local address 127.11.109.62:41669)
0621 02:55:40.920304 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:40.920310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:40.920680 (+ 370us) server_negotiation.cc:407] Connection header received
0621 02:55:40.920836 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:40.920841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:40.920926 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:40.921033 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:40.921568 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:40.922737 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:40.924098 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:40.924829 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:40.924965 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:40.926044 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:40.926050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:40.926055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:40.926075 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:40.926124 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:40.926144 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:40.926271 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:40.926420 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 02:55:40.926469 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:55:40.929026 1139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabcd8cc5434a85a3be49cd14081a80" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41669 } }
I20260621 02:55:40.929445 1139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:40.929562 1139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eabcd8cc5434a85a3be49cd14081a80, State: Initialized, Role: FOLLOWER
I20260621 02:55:40.930213 1139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabcd8cc5434a85a3be49cd14081a80" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41669 } }
I20260621 02:55:40.930473 1139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:40.930621 1139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:40.930796 1139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:40.932881 1139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabcd8cc5434a85a3be49cd14081a80" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41669 } }
I20260621 02:55:40.933430 1139 leader_election.cc:302] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eabcd8cc5434a85a3be49cd14081a80; no voters:
I20260621 02:55:40.934062 1139 leader_election.cc:288] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:40.934162 1149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:40.935163 1149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [term 1 LEADER]: Becoming Leader. State: Replica: 6eabcd8cc5434a85a3be49cd14081a80, State: Running, Role: LEADER
I20260621 02:55:40.935725 1149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabcd8cc5434a85a3be49cd14081a80" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41669 } }
I20260621 02:55:40.936398 1139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:40.938459 1150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eabcd8cc5434a85a3be49cd14081a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabcd8cc5434a85a3be49cd14081a80" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41669 } } }
I20260621 02:55:40.938814 1150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:40.939477 1151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eabcd8cc5434a85a3be49cd14081a80. Latest consensus state: current_term: 1 leader_uuid: "6eabcd8cc5434a85a3be49cd14081a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabcd8cc5434a85a3be49cd14081a80" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41669 } } }
I20260621 02:55:40.939898 1151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:40.941056 1157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:40.948390 1157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:40.956676 1157 catalog_manager.cc:1347] Generated new cluster ID: e3c01d4984a747e7be56771e9b41fee7
I20260621 02:55:40.956785 1157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:40.965816 1157 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:40.967522 1157 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:40.974224 1157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eabcd8cc5434a85a3be49cd14081a80: Generated new TSK 0
I20260621 02:55:40.975246 1157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:40.982055 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1072
W20260621 02:55:40.983371 1134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38891
W20260621 02:55:40.984069 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1072: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:40.984203 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1072
W20260621 02:55:41.001260 1057 connection.cc:535] client connection to 127.11.109.62:41669 recv error: Network error: failed to read from TLS socket (remote: 127.11.109.62:41669): Connection reset by peer (error 104)
2026-06-21T02:55:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:41 dist-test-slave-bc52 krb5kdc[1170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:41 dist-test-slave-bc52 krb5kdc[1170](info): set up 2 sockets
Jun 21 02:55:41 dist-test-slave-bc52 krb5kdc[1170](info): commencing operation
krb5kdc: starting...
W20260621 02:55:42.077405 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:42 dist-test-slave-bc52 krb5kdc[1170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:42.312992 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37101
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:37101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:42.628986 1183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:42.629292 1183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:42.629479 1183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:42.636441 1183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:42.636535 1183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:42.636581 1183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:42.636632 1183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:42.636700 1183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:42.646193 1183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:37101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:37101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:42.647382 1183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:42.648648 1183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:42.660058 1188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:42.660028 1189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:42.660028 1191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:42.660996 1183 server_base.cc:1034] running on GCE node
I20260621 02:55:42.661800 1183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:42.663002 1183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:42.664201 1183 hybrid_clock.cc:648] HybridClock initialized: now 1782010542664153 us; error 38 us; skew 500 ppm
I20260621 02:55:42.664530 1183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:42 dist-test-slave-bc52 krb5kdc[1170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010542, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:42.667567 1183 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:42.668849 1183 webserver.cc:460] Webserver started at http://127.11.109.62:35501/ using document root <none> and password file <none>
I20260621 02:55:42.669523 1183 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:42.669625 1183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:42.669857 1183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:42.672590 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "d004fa66de634b78a3f958e70a7da72d"
format_stamp: "Formatted at 2026-06-21 02:55:42 on dist-test-slave-bc52"
server_key: "c53fd2ab7cdb8944b61d315ebc1733dd"
server_key_iv: "b1a3c315765c639cc3f391cb0b66f097"
server_key_version: "encryptionkey@0"
I20260621 02:55:42.673346 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "d004fa66de634b78a3f958e70a7da72d"
format_stamp: "Formatted at 2026-06-21 02:55:42 on dist-test-slave-bc52"
server_key: "c53fd2ab7cdb8944b61d315ebc1733dd"
server_key_iv: "b1a3c315765c639cc3f391cb0b66f097"
server_key_version: "encryptionkey@0"
I20260621 02:55:42.678242 1183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:55:42.681438 1198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:42.682686 1183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:55:42.682861 1183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "d004fa66de634b78a3f958e70a7da72d"
format_stamp: "Formatted at 2026-06-21 02:55:42 on dist-test-slave-bc52"
server_key: "c53fd2ab7cdb8944b61d315ebc1733dd"
server_key_iv: "b1a3c315765c639cc3f391cb0b66f097"
server_key_version: "encryptionkey@0"
I20260621 02:55:42.683048 1183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:42.688546 1183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:42.692209 1183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:42.692498 1183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:42.709412 1183 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:37101
I20260621 02:55:42.709436 1249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:37101 every 8 connection(s)
I20260621 02:55:42.710729 1183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:42.714175 1250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:42.714808 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1183
I20260621 02:55:42.714978 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:42.715411 11700 external_mini_cluster.cc:1428] Setting key ef15f88156f1a36e9c371b74963d19f7
I20260621 02:55:42.724067 1253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:42.717027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60545 (local address 127.11.109.62:37101)
0621 02:55:42.717804 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:42.717818 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:42.717866 (+ 48us) server_negotiation.cc:407] Connection header received
0621 02:55:42.720622 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:42.720647 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:42.721060 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:42.721510 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:42.722358 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:42.722366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:42.722381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:42.722401 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:42.722469 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:42.722493 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:42.722686 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:42.722803 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 02:55:42.722966 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":346,"threads_started":1}
I20260621 02:55:42.727535 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:42.727770 1250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d: Bootstrap starting.
I20260621 02:55:42.731364 1250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:42.732609 1250 log.cc:824] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:42.735005 1253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:42.732556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59851 (local address 127.11.109.62:37101)
0621 02:55:42.732730 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:42.732736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:42.733147 (+ 411us) server_negotiation.cc:407] Connection header received
0621 02:55:42.733363 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:42.733368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:42.733411 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:42.733514 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:42.734402 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:42.734407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:42.734411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:42.734428 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:42.734473 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:42.734487 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:42.734604 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:42.734696 (+ 92us) server_negotiation.cc:299] Negotiation successful
0621 02:55:42.734750 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:55:42.735302 1250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d: No bootstrap required, opened a new log
I20260621 02:55:42.742560 1250 raft_consensus.cc:348] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d004fa66de634b78a3f958e70a7da72d" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37101 } }
I20260621 02:55:42.742911 1250 raft_consensus.cc:374] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:42.743021 1250 raft_consensus.cc:729] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d004fa66de634b78a3f958e70a7da72d, State: Initialized, Role: FOLLOWER
I20260621 02:55:42.743515 1250 consensus_queue.cc:260] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d004fa66de634b78a3f958e70a7da72d" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37101 } }
I20260621 02:55:42.743746 1250 raft_consensus.cc:388] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:42.743885 1250 raft_consensus.cc:482] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:42.744055 1250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:42.746434 1250 raft_consensus.cc:504] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d004fa66de634b78a3f958e70a7da72d" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37101 } }
I20260621 02:55:42.746893 1250 leader_election.cc:302] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d004fa66de634b78a3f958e70a7da72d; no voters:
I20260621 02:55:42.747411 1250 leader_election.cc:288] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:42.747537 1260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:42.748178 1260 raft_consensus.cc:686] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [term 1 LEADER]: Becoming Leader. State: Replica: d004fa66de634b78a3f958e70a7da72d, State: Running, Role: LEADER
I20260621 02:55:42.748652 1260 consensus_queue.cc:237] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d004fa66de634b78a3f958e70a7da72d" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37101 } }
I20260621 02:55:42.749382 1250 sys_catalog.cc:564] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:42.751186 1261 sys_catalog.cc:455] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d004fa66de634b78a3f958e70a7da72d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d004fa66de634b78a3f958e70a7da72d" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37101 } } }
I20260621 02:55:42.751605 1261 sys_catalog.cc:458] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:42.751719 1262 sys_catalog.cc:455] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d004fa66de634b78a3f958e70a7da72d. Latest consensus state: current_term: 1 leader_uuid: "d004fa66de634b78a3f958e70a7da72d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d004fa66de634b78a3f958e70a7da72d" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 37101 } } }
I20260621 02:55:42.752064 1262 sys_catalog.cc:458] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:42.753877 1265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:42.759968 1265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:42.767621 1265 catalog_manager.cc:1347] Generated new cluster ID: 2f8cb86dcba042aca9a4e0e44e307020
I20260621 02:55:42.767735 1265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:42.814131 1265 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:42.814570 1265 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:42.821102 1265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d004fa66de634b78a3f958e70a7da72d: Generated new TSK 0
I20260621 02:55:42.822016 1265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:42.839942 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1183
W20260621 02:55:42.840929 1245 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60545
W20260621 02:55:42.841643 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1183: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:42.841790 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1183
2026-06-21T02:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1857 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:42 dist-test-slave-bc52 krb5kdc[1281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:42 dist-test-slave-bc52 krb5kdc[1281](info): set up 2 sockets
Jun 21 02:55:42 dist-test-slave-bc52 krb5kdc[1281](info): commencing operation
krb5kdc: starting...
W20260621 02:55:43.929653 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.013s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:44 dist-test-slave-bc52 krb5kdc[1281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:44.160583 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:33879
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:44.480297 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:44.480574 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:44.480744 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:44.488519 1294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:44.488613 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:44.488723 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:44.488816 1294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:44.488890 1294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:44.498147 1294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:33879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:44.499306 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:44.500504 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:44.512127 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:44.512080 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:44.512161 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:44.512578 1294 server_base.cc:1034] running on GCE node
I20260621 02:55:44.513495 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:44.514744 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:44.515934 1294 hybrid_clock.cc:648] HybridClock initialized: now 1782010544515892 us; error 30 us; skew 500 ppm
I20260621 02:55:44.516273 1294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:44 dist-test-slave-bc52 krb5kdc[1281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010544, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:44.522363 1294 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:44.523624 1294 webserver.cc:460] Webserver started at http://127.11.109.62:40459/ using document root <none> and password file <none>
I20260621 02:55:44.524250 1294 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:44.524369 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:44.524602 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:44.527448 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "266053d14d6c4f8ea7068cfcb3a29283"
format_stamp: "Formatted at 2026-06-21 02:55:44 on dist-test-slave-bc52"
server_key: "d9a28cccb6969aea66e39b2d082f1608"
server_key_iv: "a85d50b770ffcaf09cf42f5820b6159b"
server_key_version: "encryptionkey@0"
I20260621 02:55:44.528113 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "266053d14d6c4f8ea7068cfcb3a29283"
format_stamp: "Formatted at 2026-06-21 02:55:44 on dist-test-slave-bc52"
server_key: "d9a28cccb6969aea66e39b2d082f1608"
server_key_iv: "a85d50b770ffcaf09cf42f5820b6159b"
server_key_version: "encryptionkey@0"
I20260621 02:55:44.532905 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260621 02:55:44.536177 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:44.537528 1294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:44.537705 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "266053d14d6c4f8ea7068cfcb3a29283"
format_stamp: "Formatted at 2026-06-21 02:55:44 on dist-test-slave-bc52"
server_key: "d9a28cccb6969aea66e39b2d082f1608"
server_key_iv: "a85d50b770ffcaf09cf42f5820b6159b"
server_key_version: "encryptionkey@0"
I20260621 02:55:44.537925 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:44.549762 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:44.553476 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:44.553759 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:44.571988 1294 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:33879
I20260621 02:55:44.572094 1360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:33879 every 8 connection(s)
I20260621 02:55:44.573376 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:44.576897 1361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:44.582350 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1294
I20260621 02:55:44.582515 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:44.582955 11700 external_mini_cluster.cc:1428] Setting key f388a6e69cbcb0c04cc9b10722053c22
I20260621 02:55:44.587419 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283: Bootstrap starting.
I20260621 02:55:44.591007 1361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:44.592237 1361 log.cc:824] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:44.595506 1361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283: No bootstrap required, opened a new log
Jun 21 02:55:44 dist-test-slave-bc52 krb5kdc[1281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:44.603820 1361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266053d14d6c4f8ea7068cfcb3a29283" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33879 } }
I20260621 02:55:44.604171 1361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:44.604291 1361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266053d14d6c4f8ea7068cfcb3a29283, State: Initialized, Role: FOLLOWER
I20260621 02:55:44.604961 1361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266053d14d6c4f8ea7068cfcb3a29283" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33879 } }
I20260621 02:55:44.605307 1361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:44.605471 1361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:44.605662 1361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:44.606225 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:44.584811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45135 (local address 127.11.109.62:33879)
0621 02:55:44.585677 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:44.585687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:44.585726 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:55:44.587323 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:44.587346 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:44.587805 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:44.588132 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:44.589932 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:44.591247 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:44.593108 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:44.593955 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:44.594092 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:44.598712 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:44.598729 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:44.598742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:44.598781 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:44.602027 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:44.602849 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:44.602858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:44.602863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:44.603034 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:44.603427 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:44.603432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:44.603437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:44.603922 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:44.604219 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:44.604676 (+ 457us) server_negotiation.cc:299] Negotiation successful
0621 02:55:44.605061 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":452,"threads_started":1}
I20260621 02:55:44.608397 1361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266053d14d6c4f8ea7068cfcb3a29283" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33879 } }
I20260621 02:55:44.608903 1361 leader_election.cc:302] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 266053d14d6c4f8ea7068cfcb3a29283; no voters:
I20260621 02:55:44.609534 1361 leader_election.cc:288] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:44.609642 1366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:44.610191 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:44.611476 1361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:44.616389 1366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [term 1 LEADER]: Becoming Leader. State: Replica: 266053d14d6c4f8ea7068cfcb3a29283, State: Running, Role: LEADER
I20260621 02:55:44.617343 1366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266053d14d6c4f8ea7068cfcb3a29283" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33879 } }
I20260621 02:55:44.620087 1367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "266053d14d6c4f8ea7068cfcb3a29283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266053d14d6c4f8ea7068cfcb3a29283" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33879 } } }
I20260621 02:55:44.620460 1367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:44.620991 1368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 266053d14d6c4f8ea7068cfcb3a29283. Latest consensus state: current_term: 1 leader_uuid: "266053d14d6c4f8ea7068cfcb3a29283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "266053d14d6c4f8ea7068cfcb3a29283" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33879 } } }
I20260621 02:55:44.621265 1368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:44.630650 1382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:44.635027 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:44.617147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41269 (local address 127.11.109.62:33879)
0621 02:55:44.617300 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:44.617313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:44.617360 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:55:44.617604 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:44.617611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:44.617681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:44.617748 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:44.620053 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:44.621230 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:44.623393 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:44.624244 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:44.624353 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:44.628052 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:44.628061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:44.628068 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:44.628121 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:44.632252 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:44.632868 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:44.632876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:44.632881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:44.633019 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:44.633440 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:44.633446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:44.633451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:44.633844 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:44.634119 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:44.634450 (+ 331us) server_negotiation.cc:299] Negotiation successful
0621 02:55:44.634678 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:55:44.636245 1382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:44.644552 1382 catalog_manager.cc:1347] Generated new cluster ID: 9701173464b745f3bbfa31b414540f8b
I20260621 02:55:44.644640 1382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:44.659242 1382 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:44.660519 1382 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:44.668813 1382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 266053d14d6c4f8ea7068cfcb3a29283: Generated new TSK 0
I20260621 02:55:44.669759 1382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:44.681357 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1294
I20260621 02:55:44.682353 1356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:44.769861 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1294
2026-06-21T02:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:44 dist-test-slave-bc52 krb5kdc[1394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:44 dist-test-slave-bc52 krb5kdc[1394](info): set up 2 sockets
Jun 21 02:55:44 dist-test-slave-bc52 krb5kdc[1394](info): commencing operation
krb5kdc: starting...
W20260621 02:55:45.873869 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:46 dist-test-slave-bc52 krb5kdc[1394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:46.108139 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:33425
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:46.425045 1407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:46.425419 1407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:46.425588 1407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:46.432581 1407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:46.432740 1407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:46.432849 1407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:46.432958 1407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:46.433043 1407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:46.442548 1407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:46.443838 1407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:46.445140 1407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:46.457743 1413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:46.457937 1412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:46.457863 1415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:46.458814 1407 server_base.cc:1034] running on GCE node
I20260621 02:55:46.459679 1407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:46.461155 1407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:46.462412 1407 hybrid_clock.cc:648] HybridClock initialized: now 1782010546462353 us; error 61 us; skew 500 ppm
I20260621 02:55:46.462761 1407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:46 dist-test-slave-bc52 krb5kdc[1394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010546, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:46.465827 1407 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:46.467284 1407 webserver.cc:460] Webserver started at http://127.11.109.62:46009/ using document root <none> and password file <none>
I20260621 02:55:46.467957 1407 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:46.468094 1407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:46.468398 1407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:46.471565 1407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "4a1add2c9ea0434c8623585e0e1132dd"
format_stamp: "Formatted at 2026-06-21 02:55:46 on dist-test-slave-bc52"
server_key: "2fee639deec39b3e6a700d2aeb0b50fb"
server_key_iv: "3b28b7a1ccadb0764a2016aa9a140a37"
server_key_version: "encryptionkey@0"
I20260621 02:55:46.472355 1407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "4a1add2c9ea0434c8623585e0e1132dd"
format_stamp: "Formatted at 2026-06-21 02:55:46 on dist-test-slave-bc52"
server_key: "2fee639deec39b3e6a700d2aeb0b50fb"
server_key_iv: "3b28b7a1ccadb0764a2016aa9a140a37"
server_key_version: "encryptionkey@0"
I20260621 02:55:46.477437 1407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:55:46.480858 1422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:46.482224 1407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:46.482404 1407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "4a1add2c9ea0434c8623585e0e1132dd"
format_stamp: "Formatted at 2026-06-21 02:55:46 on dist-test-slave-bc52"
server_key: "2fee639deec39b3e6a700d2aeb0b50fb"
server_key_iv: "3b28b7a1ccadb0764a2016aa9a140a37"
server_key_version: "encryptionkey@0"
I20260621 02:55:46.482596 1407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:46.501905 1407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:46.505743 1407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:46.506035 1407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:46.523435 1407 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:33425
I20260621 02:55:46.523443 1473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:33425 every 8 connection(s)
I20260621 02:55:46.524750 1407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:46.528409 1474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:46.530607 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1407
I20260621 02:55:46.530798 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:46.531239 11700 external_mini_cluster.cc:1428] Setting key 05c449b7c4e9b114405a2700c1217ad1
I20260621 02:55:46.539657 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd: Bootstrap starting.
I20260621 02:55:46.542928 1474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:46.543890 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:46.533102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40793 (local address 127.11.109.62:33425)
0621 02:55:46.533892 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:46.533903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:46.533946 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:55:46.535583 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:46.535605 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:46.535993 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:46.536361 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:46.536908 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:46.538285 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:46.539552 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:46.540261 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:46.540447 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:46.541682 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:46.541689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:46.541702 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:46.542219 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:46.542291 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:46.542314 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:46.542437 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:46.542671 (+ 234us) server_negotiation.cc:299] Negotiation successful
0621 02:55:46.542931 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":339,"threads_started":1}
I20260621 02:55:46.544073 1474 log.cc:824] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:46.547164 1474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd: No bootstrap required, opened a new log
I20260621 02:55:46.547531 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:46.557042 1474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1add2c9ea0434c8623585e0e1132dd" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33425 } }
I20260621 02:55:46.557571 1474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:46.557675 1474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1add2c9ea0434c8623585e0e1132dd, State: Initialized, Role: FOLLOWER
I20260621 02:55:46.558324 1474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1add2c9ea0434c8623585e0e1132dd" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33425 } }
I20260621 02:55:46.558609 1474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:46.558761 1474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:46.558941 1474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:46.559177 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:46.552633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58957 (local address 127.11.109.62:33425)
0621 02:55:46.552816 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:46.552827 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:46.553235 (+ 408us) server_negotiation.cc:407] Connection header received
0621 02:55:46.553455 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:46.553463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:46.553548 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:46.553654 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:46.554092 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:46.555197 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:46.556549 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:46.557227 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:46.557349 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:46.558393 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:46.558399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:46.558403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:46.558423 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:46.558467 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:46.558488 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:46.558587 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:46.558737 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 02:55:46.558831 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:55:46.561691 1474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1add2c9ea0434c8623585e0e1132dd" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33425 } }
I20260621 02:55:46.562114 1474 leader_election.cc:302] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a1add2c9ea0434c8623585e0e1132dd; no voters:
I20260621 02:55:46.562647 1474 leader_election.cc:288] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:46.562810 1484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:46.563454 1484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [term 1 LEADER]: Becoming Leader. State: Replica: 4a1add2c9ea0434c8623585e0e1132dd, State: Running, Role: LEADER
I20260621 02:55:46.564079 1484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1add2c9ea0434c8623585e0e1132dd" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33425 } }
I20260621 02:55:46.564596 1474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:46.566764 1486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a1add2c9ea0434c8623585e0e1132dd. Latest consensus state: current_term: 1 leader_uuid: "4a1add2c9ea0434c8623585e0e1132dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1add2c9ea0434c8623585e0e1132dd" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33425 } } }
I20260621 02:55:46.567122 1486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:46.567479 1485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a1add2c9ea0434c8623585e0e1132dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1add2c9ea0434c8623585e0e1132dd" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 33425 } } }
I20260621 02:55:46.568048 1485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:46.569052 1489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:46.576614 1489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:46.585839 1489 catalog_manager.cc:1347] Generated new cluster ID: fab7a43671e74006b9f37da5a0cb5445
I20260621 02:55:46.585963 1489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:46.604022 1489 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:46.605417 1489 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:46.614774 1489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a1add2c9ea0434c8623585e0e1132dd: Generated new TSK 0
I20260621 02:55:46.615823 1489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:46.627825 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1407
W20260621 02:55:46.628947 1469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40793
W20260621 02:55:46.629750 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1407: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:46.629905 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1407
2026-06-21T02:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1859 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:46 dist-test-slave-bc52 krb5kdc[1505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:46 dist-test-slave-bc52 krb5kdc[1505](info): set up 2 sockets
Jun 21 02:55:46 dist-test-slave-bc52 krb5kdc[1505](info): commencing operation
krb5kdc: starting...
W20260621 02:55:47.725704 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:47 dist-test-slave-bc52 krb5kdc[1505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:47.971760 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:38493
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:48.287647 1518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:48.287895 1518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:48.287983 1518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:48.295017 1518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:48.295120 1518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:48.295166 1518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:48.295244 1518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:48.295289 1518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:48.304530 1518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:48.305682 1518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:48.307375 1518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:48.318961 1523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:48.318994 1526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:48.319041 1524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:48.319779 1518 server_base.cc:1034] running on GCE node
I20260621 02:55:48.320539 1518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:48.321931 1518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:48.323179 1518 hybrid_clock.cc:648] HybridClock initialized: now 1782010548323153 us; error 41 us; skew 500 ppm
I20260621 02:55:48.323505 1518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:48 dist-test-slave-bc52 krb5kdc[1505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010548, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:48.326630 1518 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:48.327916 1518 webserver.cc:460] Webserver started at http://127.11.109.62:45957/ using document root <none> and password file <none>
I20260621 02:55:48.328505 1518 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:48.328603 1518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:48.328832 1518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:48.331707 1518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "db1648469ac04697aee0874d4d556cbb"
format_stamp: "Formatted at 2026-06-21 02:55:48 on dist-test-slave-bc52"
server_key: "99a4defbe50c30557ac0ba44b5005dfb"
server_key_iv: "2c9f1d419fcfadf37dd536dd267ebe5a"
server_key_version: "encryptionkey@0"
I20260621 02:55:48.332381 1518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "db1648469ac04697aee0874d4d556cbb"
format_stamp: "Formatted at 2026-06-21 02:55:48 on dist-test-slave-bc52"
server_key: "99a4defbe50c30557ac0ba44b5005dfb"
server_key_iv: "2c9f1d419fcfadf37dd536dd267ebe5a"
server_key_version: "encryptionkey@0"
I20260621 02:55:48.337329 1518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:55:48.340476 1533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:48.341763 1518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:48.341897 1518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "db1648469ac04697aee0874d4d556cbb"
format_stamp: "Formatted at 2026-06-21 02:55:48 on dist-test-slave-bc52"
server_key: "99a4defbe50c30557ac0ba44b5005dfb"
server_key_iv: "2c9f1d419fcfadf37dd536dd267ebe5a"
server_key_version: "encryptionkey@0"
I20260621 02:55:48.342060 1518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:48.347293 1518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:48.351317 1518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:48.351554 1518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:48.368716 1518 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:38493
I20260621 02:55:48.368748 1584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:38493 every 8 connection(s)
I20260621 02:55:48.370071 1518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:48.373590 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:48.375165 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1518
I20260621 02:55:48.375346 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:48.375788 11700 external_mini_cluster.cc:1428] Setting key b38ef4d1cf261a7f50ea906e9f2a77d1
I20260621 02:55:48.383790 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:48.377555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40775 (local address 127.11.109.62:38493)
0621 02:55:48.378372 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:48.378386 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:48.378438 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:55:48.380139 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:48.380170 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:48.380578 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:48.381014 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:48.382012 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:48.382019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:48.382035 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:48.382052 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:48.382117 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:48.382148 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:48.382350 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:48.382469 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 02:55:48.382652 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260621 02:55:48.387141 1585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb: Bootstrap starting.
I20260621 02:55:48.387326 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:48.390758 1585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:48.392014 1585 log.cc:824] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:48.394709 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:48.392335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57507 (local address 127.11.109.62:38493)
0621 02:55:48.392562 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:48.392567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:48.392936 (+ 369us) server_negotiation.cc:407] Connection header received
0621 02:55:48.393047 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:48.393055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:48.393111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:48.393275 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:48.394084 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:48.394087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:48.394091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:55:48.394104 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:48.394139 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:55:48.394152 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:48.394233 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:48.394374 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 02:55:48.394414 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:55:48.395321 1585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb: No bootstrap required, opened a new log
I20260621 02:55:48.402505 1585 raft_consensus.cc:348] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db1648469ac04697aee0874d4d556cbb" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38493 } }
I20260621 02:55:48.402938 1585 raft_consensus.cc:374] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:55:48.403041 1585 raft_consensus.cc:729] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1648469ac04697aee0874d4d556cbb, State: Initialized, Role: FOLLOWER
I20260621 02:55:48.403640 1585 consensus_queue.cc:260] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db1648469ac04697aee0874d4d556cbb" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38493 } }
I20260621 02:55:48.403865 1585 raft_consensus.cc:388] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:55:48.404009 1585 raft_consensus.cc:482] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:55:48.404194 1585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:48.406296 1585 raft_consensus.cc:504] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db1648469ac04697aee0874d4d556cbb" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38493 } }
I20260621 02:55:48.406764 1585 leader_election.cc:302] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db1648469ac04697aee0874d4d556cbb; no voters:
I20260621 02:55:48.407296 1585 leader_election.cc:288] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:55:48.407440 1595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:48.408098 1595 raft_consensus.cc:686] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [term 1 LEADER]: Becoming Leader. State: Replica: db1648469ac04697aee0874d4d556cbb, State: Running, Role: LEADER
I20260621 02:55:48.408623 1595 consensus_queue.cc:237] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db1648469ac04697aee0874d4d556cbb" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38493 } }
I20260621 02:55:48.409276 1585 sys_catalog.cc:564] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:48.411484 1597 sys_catalog.cc:455] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader db1648469ac04697aee0874d4d556cbb. Latest consensus state: current_term: 1 leader_uuid: "db1648469ac04697aee0874d4d556cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db1648469ac04697aee0874d4d556cbb" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38493 } } }
I20260621 02:55:48.411433 1596 sys_catalog.cc:455] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db1648469ac04697aee0874d4d556cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db1648469ac04697aee0874d4d556cbb" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 38493 } } }
I20260621 02:55:48.411880 1597 sys_catalog.cc:458] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:48.411899 1596 sys_catalog.cc:458] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:48.413816 1600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:48.418635 1600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:48.427546 1600 catalog_manager.cc:1347] Generated new cluster ID: f0066c97701f4e6193024a4bc3a69553
I20260621 02:55:48.427662 1600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:48.442835 1600 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:48.443243 1600 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:48.455487 1600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db1648469ac04697aee0874d4d556cbb: Generated new TSK 0
I20260621 02:55:48.456252 1600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:48.463248 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1518
W20260621 02:55:48.464435 1580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40775
W20260621 02:55:48.465219 11700 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1518: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:55:48.465386 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1518
2026-06-21T02:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1832 ms)
[----------] 18 tests from AuthTokenIssuingTest (33835 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:48 dist-test-slave-bc52 krb5kdc[1616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:48 dist-test-slave-bc52 krb5kdc[1616](info): set up 2 sockets
Jun 21 02:55:48 dist-test-slave-bc52 krb5kdc[1616](info): commencing operation
krb5kdc: starting...
W20260621 02:55:49.565889 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:49 dist-test-slave-bc52 krb5kdc[1616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:49.804626 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35227
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35227,127.11.109.61:33445,127.11.109.60:46211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:50.119180 1629 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:55:50.119491 1629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:50.119563 1629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:50.119635 1629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:50.126493 1629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:50.126576 1629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:50.126631 1629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:50.126679 1629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:50.126732 1629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:50.135854 1629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35227,127.11.109.61:33445,127.11.109.60:46211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:50.137054 1629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:50.138314 1629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:50.149567 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:50.149602 1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:50.149799 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:50.150300 1629 server_base.cc:1034] running on GCE node
I20260621 02:55:50.150959 1629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:50.152231 1629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:50.153462 1629 hybrid_clock.cc:648] HybridClock initialized: now 1782010550153395 us; error 48 us; skew 500 ppm
I20260621 02:55:50.153795 1629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:50 dist-test-slave-bc52 krb5kdc[1616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:50.159900 1629 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:50.161345 1629 webserver.cc:460] Webserver started at http://127.11.109.62:37213/ using document root <none> and password file <none>
I20260621 02:55:50.162014 1629 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:50.162142 1629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:50.162464 1629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:50.165333 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8"
format_stamp: "Formatted at 2026-06-21 02:55:50 on dist-test-slave-bc52"
server_key: "b8ca313443f7715485c7576d14179b98"
server_key_iv: "eab96e5192077d2f2a9d976517f95d1f"
server_key_version: "encryptionkey@0"
I20260621 02:55:50.166121 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8"
format_stamp: "Formatted at 2026-06-21 02:55:50 on dist-test-slave-bc52"
server_key: "b8ca313443f7715485c7576d14179b98"
server_key_iv: "eab96e5192077d2f2a9d976517f95d1f"
server_key_version: "encryptionkey@0"
I20260621 02:55:50.171022 1629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:55:50.174258 1644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:50.175441 1629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:55:50.175616 1629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8"
format_stamp: "Formatted at 2026-06-21 02:55:50 on dist-test-slave-bc52"
server_key: "b8ca313443f7715485c7576d14179b98"
server_key_iv: "eab96e5192077d2f2a9d976517f95d1f"
server_key_version: "encryptionkey@0"
I20260621 02:55:50.175820 1629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:50.196358 1629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:50.199994 1629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:50.200275 1629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:50.218176 1629 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:35227
I20260621 02:55:50.218209 1695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:35227 every 8 connection(s)
I20260621 02:55:50.219380 1629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:50.222891 1696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:50.226290 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1629
I20260621 02:55:50.226495 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:50.227108 11700 external_mini_cluster.cc:1428] Setting key 92e01b1e69dd5b7eafed7d473e3db1b2
I20260621 02:55:50.229271 1696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:50 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.61@KRBTEST.COM; defaulting to no policy
I20260621 02:55:50.250273 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.232191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.11.109.62:35227)
0621 02:55:50.233791 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:50.233799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:50.233850 (+ 51us) server_negotiation.cc:407] Connection header received
0621 02:55:50.235319 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:50.235324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:50.235532 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:50.235760 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:50.237031 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.238242 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.239570 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.240091 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.240285 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.243451 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:50.243458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:50.243468 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:50.243498 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:50.246223 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.246922 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.246930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.246934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.247086 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.247466 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.247472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.247480 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.247894 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:50.248109 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:50.248598 (+ 489us) server_negotiation.cc:299] Negotiation successful
0621 02:55:50.248900 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1388,"thread_start_us":1175,"threads_started":1}
I20260621 02:55:50.250313 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.232190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35227 (local address 127.0.0.1:47340)
0621 02:55:50.232997 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.233056 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:50.234960 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:50.235770 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:50.235777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:50.236488 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:50.236833 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.236849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.238401 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.238410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.239420 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.239432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.240315 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.240322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.240501 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.241128 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:50.241150 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:50.243237 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:50.246397 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.246407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.246417 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.246795 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.247195 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.247201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.247205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.247354 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.248042 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:50.248049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:50.248250 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 02:55:50.248484 (+ 234us) client_negotiation.cc:240] Negotiation successful
0621 02:55:50.249012 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":136,"thread_start_us":364,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.61@KRBTEST.COM" created.
I20260621 02:55:50.253038 1696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:50.254103 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.253629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44586)
0621 02:55:50.253824 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.253943 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 02:55:50.254345 1647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
I20260621 02:55:50.255877 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.255443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44598)
0621 02:55:50.255606 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.255695 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:55:50.256089 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:33445: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab.
Entry for principal kudu/127.11.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab.
I20260621 02:55:50.278347 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:33445
--webserver_interface=127.11.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab
--principal=kudu/127.11.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35227,127.11.109.61:33445,127.11.109.60:46211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:55:50.305428 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } attempt: 1
I20260621 02:55:50.306545 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.306109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44604)
0621 02:55:50.306293 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.306412 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:55:50.307451 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.307131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44618)
0621 02:55:50.307266 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.307334 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:55:50.307654 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:33445: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
I20260621 02:55:50.375927 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } attempt: 2
I20260621 02:55:50.377128 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.376633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44622)
0621 02:55:50.376894 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.376977 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 02:55:50.378159 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.377800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44630)
0621 02:55:50.377972 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.378033 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:55:50.378392 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:33445: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
I20260621 02:55:50.469674 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } attempt: 3
I20260621 02:55:50.471015 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.470509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44642)
0621 02:55:50.470735 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.470843 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 02:55:50.472143 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.471814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44650)
0621 02:55:50.471949 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.472014 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 02:55:50.472323 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:33445: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
W20260621 02:55:50.594501 1703 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:55:50.594847 1703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:50.594992 1703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:50.595129 1703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:50.601867 1703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:50.601979 1703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:50.602083 1703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:50.602180 1703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:50.602267 1703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:50.611901 1703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35227,127.11.109.61:33445,127.11.109.60:46211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab
--principal=kudu/127.11.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:50.613106 1703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:50.614408 1703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:55:50.615535 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } attempt: 4
I20260621 02:55:50.616703 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.616217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44666)
0621 02:55:50.616496 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.616580 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260621 02:55:50.617749 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.617414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44670)
0621 02:55:50.617554 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.617637 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:55:50.617930 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:33445: Network error: Client connection negotiation failed: client connection to 127.11.109.61:33445: connect: Connection refused (error 111)
W20260621 02:55:50.626685 1708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:50.626735 1711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:50.626745 1709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:50.627626 1703 server_base.cc:1034] running on GCE node
I20260621 02:55:50.628350 1703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:50.629666 1703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:50.630873 1703 hybrid_clock.cc:648] HybridClock initialized: now 1782010550630816 us; error 29 us; skew 500 ppm
I20260621 02:55:50.631141 1703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:50 dist-test-slave-bc52 krb5kdc[1616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:50.634045 1703 init.cc:377] Logged in from keytab as kudu/127.11.109.61@KRBTEST.COM (short username kudu)
I20260621 02:55:50.635345 1703 webserver.cc:460] Webserver started at http://127.11.109.61:37151/ using document root <none> and password file <none>
I20260621 02:55:50.635962 1703 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:50.636045 1703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:50.636291 1703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:50.639081 1703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data/instance:
uuid: "92254225c45b474a9a8da5f25017e440"
format_stamp: "Formatted at 2026-06-21 02:55:50 on dist-test-slave-bc52"
server_key: "6ce8a4fd9bdde6f4f3c9b659112e6ae4"
server_key_iv: "59f344f60057a637f243548421724a7f"
server_key_version: "encryptionkey@0"
I20260621 02:55:50.639796 1703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal/instance:
uuid: "92254225c45b474a9a8da5f25017e440"
format_stamp: "Formatted at 2026-06-21 02:55:50 on dist-test-slave-bc52"
server_key: "6ce8a4fd9bdde6f4f3c9b659112e6ae4"
server_key_iv: "59f344f60057a637f243548421724a7f"
server_key_version: "encryptionkey@0"
I20260621 02:55:50.644500 1703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:55:50.647812 1718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:50.649021 1703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:55:50.649228 1703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal
uuid: "92254225c45b474a9a8da5f25017e440"
format_stamp: "Formatted at 2026-06-21 02:55:50 on dist-test-slave-bc52"
server_key: "6ce8a4fd9bdde6f4f3c9b659112e6ae4"
server_key_iv: "59f344f60057a637f243548421724a7f"
server_key_version: "encryptionkey@0"
I20260621 02:55:50.649451 1703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:50.662744 1703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:50.666958 1703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:50.667222 1703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:50.685019 1703 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.61:33445
I20260621 02:55:50.685043 1769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.61:33445 every 8 connection(s)
I20260621 02:55:50.686352 1703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
I20260621 02:55:50.689868 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:50.691363 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1703
I20260621 02:55:50.691537 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-1/wal/instance
I20260621 02:55:50.692039 11700 external_mini_cluster.cc:1428] Setting key 46c28ed7b1f7ccded9e39c733b0440ce
I20260621 02:55:50.698858 1770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:50 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.60@KRBTEST.COM; defaulting to no policy
I20260621 02:55:50.722265 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.702087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.11.109.62:35227)
0621 02:55:50.702286 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:50.702293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:50.703426 (+ 1133us) server_negotiation.cc:407] Connection header received
0621 02:55:50.706291 (+ 2865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:50.706295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:50.706361 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:50.706451 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:50.708452 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.709097 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.711477 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.711875 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.711964 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.716400 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:50.716406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:50.716410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:50.716455 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:50.719157 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.720109 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.720113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.720115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.720263 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.720688 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.720691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.720693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.720941 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:50.721045 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:50.721761 (+ 716us) server_negotiation.cc:299] Negotiation successful
0621 02:55:50.721919 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.60@KRBTEST.COM" created.
I20260621 02:55:50.723167 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.702486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35227 (local address 127.0.0.1:47354)
0621 02:55:50.703225 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.703277 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:50.706006 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:50.706722 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:50.706734 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:50.707462 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:50.708210 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.708235 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.709273 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.709281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.711317 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.711335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.712103 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.712113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.712435 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.713205 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:50.713241 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:50.716151 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:50.719306 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.719314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.719332 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.719981 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.720362 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.720368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.720373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.720588 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.721050 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:50.721056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:50.721427 (+ 371us) client_negotiation.cc:769] Sending connection context
0621 02:55:50.721651 (+ 224us) client_negotiation.cc:240] Negotiation successful
0621 02:55:50.721991 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":324,"threads_started":1}
I20260621 02:55:50.724838 1770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:50 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:50.739235 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.725882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.11.109.61:33445)
0621 02:55:50.726471 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:50.726477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:50.726499 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:55:50.726537 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:50.726542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:50.726712 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:50.726865 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:50.727543 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.728601 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.729750 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.730247 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.730371 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.733610 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:50.733619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:50.733630 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:50.733657 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:50.736377 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.737227 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.737237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.737244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.737350 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.737771 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.737774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.737776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.738078 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:50.738231 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:50.738646 (+ 415us) server_negotiation.cc:299] Negotiation successful
0621 02:55:50.738959 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":451,"thread_start_us":327,"threads_started":1}
I20260621 02:55:50.739344 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.725695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44682)
0621 02:55:50.725865 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.725884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:50.725977 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:50.726849 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:50.726853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:50.727266 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:50.727381 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.727388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.728728 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.728736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.729623 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.729634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.730389 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.730395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.730572 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.731095 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:50.731129 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:50.733416 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:50.736502 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.736510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.736517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.737058 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.737466 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.737474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.737480 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.737666 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.738203 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:50.738211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:50.738371 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 02:55:50.738529 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:55:50.738853 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:55:50.741706 1770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:50.742813 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.742286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58604)
0621 02:55:50.742481 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.742625 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:55:50.743067 1722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:50.744696 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.744296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58618)
0621 02:55:50.744464 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.744565 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 02:55:50.744927 1770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab.
Entry for principal kudu/127.11.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab.
I20260621 02:55:50.753013 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:46211
--webserver_interface=127.11.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab
--principal=kudu/127.11.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:35227,127.11.109.61:33445,127.11.109.60:46211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:55:50.794220 1770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 1
I20260621 02:55:50.795434 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.794903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58624)
0621 02:55:50.795137 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.795261 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:55:50.796533 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.796036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58634)
0621 02:55:50.796262 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.796364 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260621 02:55:50.796726 1770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:50.864981 1770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 2
I20260621 02:55:50.866238 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.865715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58646)
0621 02:55:50.865930 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.866041 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:55:50.867429 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.866991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58654)
0621 02:55:50.867167 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.867265 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:55:50.867702 1770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:50.917166 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } attempt: 5
Jun 21 02:55:50 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:50.930269 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.918054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.11.109.61:33445)
0621 02:55:50.918225 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:50.918233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:50.918293 (+ 60us) server_negotiation.cc:407] Connection header received
0621 02:55:50.918512 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:50.918516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:50.918600 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:50.918706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:50.919478 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.920171 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.921369 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.921799 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.921941 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.925124 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:50.925129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:50.925132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:50.925162 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:50.927774 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.928396 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.928408 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.928411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.928526 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:50.928890 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:50.928896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:50.928899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:50.929162 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:50.929362 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:50.929730 (+ 368us) server_negotiation.cc:299] Negotiation successful
0621 02:55:50.929955 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:50.930284 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.917971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44686)
0621 02:55:50.918192 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.918213 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:50.918358 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:50.918741 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:50.918745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:50.919160 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:50.919333 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.919341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.920302 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.920308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.921164 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:50.921226 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:50.921961 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:50.921967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:50.922183 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:50.922789 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:50.922807 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:50.924924 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:50.927911 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.927917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.927921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.928276 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.928641 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:50.928645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:50.928649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:50.928784 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:50.929326 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:50.929330 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:50.929489 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 02:55:50.929627 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:55:50.929915 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 02:55:50.931521 1696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:50.932440 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.932087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58658)
0621 02:55:50.932237 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.932315 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260621 02:55:50.933382 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.932996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58660)
0621 02:55:50.933155 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.933288 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:55:50.933600 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:50.958899 1770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 3
I20260621 02:55:50.960107 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.959458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58676)
0621 02:55:50.959692 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.959837 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 02:55:50.961270 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.960787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58686)
0621 02:55:50.961014 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.961112 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260621 02:55:50.961504 1770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:50.984794 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 1
I20260621 02:55:50.985976 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.985437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58696)
0621 02:55:50.985669 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.985780 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:55:50.987042 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:50.986627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58706)
0621 02:55:50.986794 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:50.986895 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 02:55:50.987268 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:51.055495 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 2
I20260621 02:55:51.056658 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.056138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58718)
0621 02:55:51.056365 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.056465 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 02:55:51.057669 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.057321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58724)
0621 02:55:51.057464 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.057544 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 02:55:51.057919 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
W20260621 02:55:51.072620 1777 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:55:51.072961 1777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:51.073078 1777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:51.073251 1777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:51.080230 1777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:51.080344 1777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:51.080453 1777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:51.080539 1777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:51.080628 1777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:51.090018 1777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:35227,127.11.109.61:33445,127.11.109.60:46211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab
--principal=kudu/127.11.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:51.091123 1777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:51.092294 1777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:51.103116 1782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:51.103336 1785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:51.103374 1783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:51.104171 1777 server_base.cc:1034] running on GCE node
I20260621 02:55:51.104696 1770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 4
I20260621 02:55:51.104816 1777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:51.105751 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.105308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58732)
0621 02:55:51.105506 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.105610 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 02:55:51.106408 1777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:51.106833 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.106432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58744)
0621 02:55:51.106606 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.106710 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260621 02:55:51.107080 1770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:51.107648 1777 hybrid_clock.cc:648] HybridClock initialized: now 1782010551107596 us; error 25 us; skew 500 ppm
I20260621 02:55:51.107918 1777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:51.110942 1777 init.cc:377] Logged in from keytab as kudu/127.11.109.60@KRBTEST.COM (short username kudu)
I20260621 02:55:51.112243 1777 webserver.cc:460] Webserver started at http://127.11.109.60:44825/ using document root <none> and password file <none>
I20260621 02:55:51.112891 1777 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:51.113023 1777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:51.113394 1777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:51.116264 1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data/instance:
uuid: "e4076bca9109495a9882b31f93df47a3"
format_stamp: "Formatted at 2026-06-21 02:55:51 on dist-test-slave-bc52"
server_key: "560521a7890ce0de4f90f4a86cf229e7"
server_key_iv: "37bd1014fd1b613bbc3c1b56cdcd2d4d"
server_key_version: "encryptionkey@0"
I20260621 02:55:51.117058 1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal/instance:
uuid: "e4076bca9109495a9882b31f93df47a3"
format_stamp: "Formatted at 2026-06-21 02:55:51 on dist-test-slave-bc52"
server_key: "560521a7890ce0de4f90f4a86cf229e7"
server_key_iv: "37bd1014fd1b613bbc3c1b56cdcd2d4d"
server_key_version: "encryptionkey@0"
I20260621 02:55:51.121891 1777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:55:51.125164 1792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:51.126462 1777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:55:51.126643 1777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal
uuid: "e4076bca9109495a9882b31f93df47a3"
format_stamp: "Formatted at 2026-06-21 02:55:51 on dist-test-slave-bc52"
server_key: "560521a7890ce0de4f90f4a86cf229e7"
server_key_iv: "37bd1014fd1b613bbc3c1b56cdcd2d4d"
server_key_version: "encryptionkey@0"
I20260621 02:55:51.126848 1777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:51.155180 1777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:51.159337 1777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:51.159605 1777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:51.164119 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 3
I20260621 02:55:51.165208 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.164685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58752)
0621 02:55:51.164869 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.164977 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:55:51.166443 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.165972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58754)
0621 02:55:51.166207 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.166308 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260621 02:55:51.166676 1696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:46211: Network error: Client connection negotiation failed: client connection to 127.11.109.60:46211: connect: Connection refused (error 111)
I20260621 02:55:51.178241 1777 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.60:46211
I20260621 02:55:51.178247 1843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.60:46211 every 8 connection(s)
I20260621 02:55:51.179539 1777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
I20260621 02:55:51.183145 1844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:51.184856 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 1777
I20260621 02:55:51.185007 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782010204918124-11700-0/minicluster-data/master-2/wal/instance
I20260621 02:55:51.185534 11700 external_mini_cluster.cc:1428] Setting key 7c2f0b8da326caf465bade8246d803cd
I20260621 02:55:51.192054 1844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:51.208356 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.187310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.11.109.60:46211)
0621 02:55:51.188106 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.188121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.188167 (+ 46us) server_negotiation.cc:407] Connection header received
0621 02:55:51.189861 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.189885 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.190321 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.190677 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.191980 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.193997 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.195462 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.196368 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.196571 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.201306 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.201318 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.201332 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.201398 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.204440 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.205214 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.205222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.205230 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.205402 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.205830 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.205833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.205835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.206389 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.206588 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.207059 (+ 471us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.207359 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":331,"threads_started":1}
I20260621 02:55:51.211189 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.194977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:35227 (local address 127.0.0.1:47366)
0621 02:55:51.195670 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.195718 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:51.195868 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:51.196230 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:51.196234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:51.196799 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:51.197028 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.197035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.198108 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.198114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.199661 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.199672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.200739 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.200744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.200857 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.201442 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:51.201470 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:51.204170 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:51.207794 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.207806 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.207812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.208263 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.209407 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.209414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.209417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.209577 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.210161 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:51.210165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:51.210304 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 02:55:51.210503 (+ 199us) client_negotiation.cc:240] Negotiation successful
0621 02:55:51.210779 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":348,"threads_started":1}
I20260621 02:55:51.211198 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.194911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.11.109.62:35227)
0621 02:55:51.195111 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.195119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.195819 (+ 700us) server_negotiation.cc:407] Connection header received
0621 02:55:51.195993 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.195999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.196075 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.196179 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.197225 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.197935 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.199831 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.200537 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.200686 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.204357 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.204361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.204364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.204394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.207673 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.208399 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.208407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.208410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.209299 (+ 889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.209688 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.209695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.209699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.210023 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.210179 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.210639 (+ 460us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.210845 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:55:51.212711 1844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:51.227046 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.211353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.11.109.62:35227)
0621 02:55:51.211507 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.211512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.211553 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:55:51.211605 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.211612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.211685 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.211787 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.213514 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.214469 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.215747 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.216145 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.216282 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.220817 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.220823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.220828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.220864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.224406 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.225138 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.225144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.225147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.225317 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.225677 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.225682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.225685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.225952 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.226236 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.226506 (+ 270us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.226727 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:55:51.227108 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.213618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.11.109.61:33445)
0621 02:55:51.213802 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.213808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.213825 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:55:51.213879 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.213885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.213956 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.214059 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.214669 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.215300 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.216698 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.217355 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.217483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.221134 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.221144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.221147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.221219 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.224122 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.224847 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.224853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.224857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.225016 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.225517 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.225524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.225527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.225896 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.226009 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.226521 (+ 512us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.226742 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:55:51.227843 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.213529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:33445 (local address 127.0.0.1:44688)
0621 02:55:51.213678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.213696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:51.213782 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:51.214080 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:51.214082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:51.214384 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:51.214500 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.214506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.215434 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.215440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.216564 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.216576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.217513 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.217519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.217702 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.218270 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:51.218290 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:51.220948 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:51.224261 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.224266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.224270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.224729 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.225159 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.225164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.225167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.225363 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.226103 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:51.226108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:51.226260 (+ 152us) client_negotiation.cc:769] Sending connection context
0621 02:55:51.227266 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0621 02:55:51.227516 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:55:51.228859 1844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:51.241477 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.229543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58772)
0621 02:55:51.229672 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.229689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:51.229790 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:51.230134 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:51.230138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:51.230470 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:51.230582 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.230588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.231622 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.231627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.232449 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.232458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.233118 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.233123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.233379 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.233972 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:51.233988 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:51.235971 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:51.239081 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.239086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.239090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.239568 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.239898 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.239903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.239906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.240087 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.240571 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:51.240576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:51.240731 (+ 155us) client_negotiation.cc:769] Sending connection context
0621 02:55:51.240849 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 02:55:51.241037 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":35}
I20260621 02:55:51.241544 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.229794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.11.109.60:46211)
0621 02:55:51.229936 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.229940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.229951 (+ 11us) server_negotiation.cc:407] Connection header received
0621 02:55:51.229979 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.229982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.230023 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.230119 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.230737 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.231487 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.232606 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.232986 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.233112 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.236175 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.236180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.236183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.236208 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.238939 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.239675 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.239679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.239681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.239791 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.240214 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.240217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.240219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.240460 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.240586 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.240984 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.241254 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
I20260621 02:55:51.243546 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.228922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.11.109.61:33445)
0621 02:55:51.229102 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.229109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.229128 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:55:51.229196 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.229202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.229270 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.229389 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.230520 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.231389 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.232582 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.232998 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.233121 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.237513 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.237519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.237524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.237561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.240492 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.241294 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.241299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.241304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.241456 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.242107 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.242114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.242120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.242457 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.242707 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.242997 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.243215 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:55:51.246753 1844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: Bootstrap starting.
I20260621 02:55:51.248196 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:51.250627 1844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:51.251824 1844 log.cc:824] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:51.254642 1844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: No bootstrap required, opened a new log
I20260621 02:55:51.251569 1671 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372
I20260621 02:55:51.261426 1844 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.261719 1844 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4076bca9109495a9882b31f93df47a3, State: Initialized, Role: FOLLOWER
I20260621 02:55:51.262130 1844 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.262964 1850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.263304 1850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.263751 1844 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:55:51.272822 1861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:55:51.272941 1861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:55:51.270567 1777 master_runner.cc:186] Error getting master registration for 127.11.109.62:35227: OK, instance_id { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" instance_seqno: 1782010550213141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:55:51.278290 1777 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:55:51.344027 1696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 4
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:55:51.360551 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.344788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58786)
0621 02:55:51.345078 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.345098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:51.345251 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:51.345709 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:51.345714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:51.346230 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:51.346434 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.346446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.347962 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.347968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.349497 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.349513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.350476 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.350483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.350700 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.351397 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:51.351420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:51.354099 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:51.357971 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.357976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.357980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.358457 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.358824 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.358828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.358832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.358979 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.359618 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:51.359623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:51.359783 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 02:55:51.359929 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 02:55:51.360191 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260621 02:55:51.360589 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.344840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.11.109.60:46211)
0621 02:55:51.345055 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.345062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.345193 (+ 131us) server_negotiation.cc:407] Connection header received
0621 02:55:51.345402 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.345427 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.345526 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.345651 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.346607 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.347807 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.349629 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.350328 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.350443 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.354292 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.354299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.354304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.354343 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.357846 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.358582 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.358588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.358592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.358719 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.359091 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.359096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.359100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.359493 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.359654 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.360060 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.360269 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:55:51.366520 1696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8: Bootstrap starting.
I20260621 02:55:51.370496 1696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:51.371748 1696 log.cc:824] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:51.374656 1696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8: No bootstrap required, opened a new log
I20260621 02:55:51.379719 1696 raft_consensus.cc:348] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.380131 1696 raft_consensus.cc:729] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e025e3d8ab7e4e88aeb090a2b563c1c8, State: Initialized, Role: FOLLOWER
I20260621 02:55:51.380550 1696 consensus_queue.cc:260] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.381807 1864 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.382418 1864 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.382954 1696 sys_catalog.cc:564] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:51.383015 1671 raft_consensus.cc:482] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 02:55:51.383189 1671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:51.387147 1671 raft_consensus.cc:504] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.389148 1671 leader_election.cc:288] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [CANDIDATE]: Term 1 election: Requested vote from peers 92254225c45b474a9a8da5f25017e440 (127.11.109.61:33445), e4076bca9109495a9882b31f93df47a3 (127.11.109.60:46211)
I20260621 02:55:51.389725 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e4076bca9109495a9882b31f93df47a3"
I20260621 02:55:51.390228 1819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:51.394289 1819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e025e3d8ab7e4e88aeb090a2b563c1c8 in term 1.
I20260621 02:55:51.394973 1648 leader_election.cc:302] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e025e3d8ab7e4e88aeb090a2b563c1c8, e4076bca9109495a9882b31f93df47a3; no voters:
I20260621 02:55:51.396766 1864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:55:51.397603 1864 raft_consensus.cc:686] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [term 1 LEADER]: Becoming Leader. State: Replica: e025e3d8ab7e4e88aeb090a2b563c1c8, State: Running, Role: LEADER
I20260621 02:55:51.398109 1864 consensus_queue.cc:237] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.403806 1872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e025e3d8ab7e4e88aeb090a2b563c1c8. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.404215 1872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:51.405570 1874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:51.406291 1770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } attempt: 5
W20260621 02:55:51.411553 1798 tablet.cc:2369] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:55:51 dist-test-slave-bc52 krb5kdc[1616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:55:51.443372 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.407027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:46211 (local address 127.0.0.1:58788)
0621 02:55:51.407468 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:51.407495 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:51.407657 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:51.408509 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:51.408517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:51.409196 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:51.409464 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.409479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.411403 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.411413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.413403 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:51.413422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.415111 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.415120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:51.415618 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.416993 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:51.417061 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:51.422502 (+ 5441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:51.431992 (+ 9490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.432002 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.432009 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.433490 (+ 1481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.434986 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:51.435018 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:51.435025 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:51.435521 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:51.438316 (+ 2795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:51.438349 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:51.438708 (+ 359us) client_negotiation.cc:769] Sending connection context
0621 02:55:51.438985 (+ 277us) client_negotiation.cc:240] Negotiation successful
0621 02:55:51.440916 (+ 1931us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260621 02:55:51.444813 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.407206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.11.109.60:46211)
0621 02:55:51.407928 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.407937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.407959 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:55:51.408025 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.408033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.408125 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.408244 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.409726 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.411194 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.413664 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.414623 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.414782 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.422910 (+ 8128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.422919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.422927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.422978 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.431483 (+ 8505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.433785 (+ 2302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.433794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.433800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.434613 (+ 813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.435979 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.435990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.435995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.437870 (+ 1875us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.439801 (+ 1931us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.441206 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.442611 (+ 1405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565}
I20260621 02:55:51.452589 1770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: Bootstrap starting.
W20260621 02:55:51.452919 1650 tablet.cc:2369] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:55:51.460575 1874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:51.461640 1770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:51.463271 1770 log.cc:824] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:51.479782 1770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: No bootstrap required, opened a new log
I20260621 02:55:51.494700 1819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [term 1 FOLLOWER]: Refusing update from remote peer e025e3d8ab7e4e88aeb090a2b563c1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:51.495733 1864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:51.503952 1770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.512064 1770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92254225c45b474a9a8da5f25017e440, State: Initialized, Role: FOLLOWER
I20260621 02:55:51.512853 1770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } }
I20260621 02:55:51.513595 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.484646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.11.109.61:33445)
0621 02:55:51.484847 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.484855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.485494 (+ 639us) server_negotiation.cc:407] Connection header received
0621 02:55:51.485782 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.485788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.485864 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.485947 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.488704 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.489949 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.491905 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.492595 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.492703 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.503873 (+ 11170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.503882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.503888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.503946 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.508283 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.509524 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.509531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.509537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.509737 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.510391 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.510397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.510402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.510825 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.511105 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.512688 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.513015 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:55:51.520282 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.525417 1889 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:55:51.522780 1770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:51.522858 1745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92254225c45b474a9a8da5f25017e440"
I20260621 02:55:51.523483 1744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:51.525712 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.530342 1744 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [term 1 FOLLOWER]: Refusing update from remote peer e025e3d8ab7e4e88aeb090a2b563c1c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:51.533802 1745 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e025e3d8ab7e4e88aeb090a2b563c1c8 in term 1.
I20260621 02:55:51.538203 1864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:51.542618 1850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e025e3d8ab7e4e88aeb090a2b563c1c8. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.542882 1850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.544528 1884 mvcc.cc:205] Tried to move back new op lower bound from 7299115218906853376 to 7299115218531377152. Current Snapshot: MvccSnapshot[applied={T|T < 7299115218906853376}]
I20260621 02:55:51.550689 1886 mvcc.cc:205] Tried to move back new op lower bound from 7299115218906853376 to 7299115218531377152. Current Snapshot: MvccSnapshot[applied={T|T < 7299115218906853376}]
I20260621 02:55:51.557214 1872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.557502 1872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:51.563263 1850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.563540 1850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.566319 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e025e3d8ab7e4e88aeb090a2b563c1c8. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.566761 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.567190 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.520831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.11.109.60:46211)
0621 02:55:51.521050 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.521058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.523819 (+ 2761us) server_negotiation.cc:407] Connection header received
0621 02:55:51.527246 (+ 3427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.527262 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.527360 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.527425 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.531277 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.532316 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.543250 (+ 10934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.543916 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.544017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.558526 (+ 14509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.558532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.558536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.558595 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.562142 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.564204 (+ 2062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.564212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.564218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.564382 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.564803 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.564807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.564810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.565273 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.565554 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.566554 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.566784 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:55:51.570796 1864 sys_catalog.cc:455] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.571280 1864 sys_catalog.cc:458] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:51.533138 1629 master_runner.cc:186] Error getting master registration for 127.11.109.61:33445: OK, instance_id { permanent_uuid: "92254225c45b474a9a8da5f25017e440" instance_seqno: 1782010550679852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:55:51.578439 1629 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:55:51.582600 1888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025e3d8ab7e4e88aeb090a2b563c1c8" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 35227 } } peers { permanent_uuid: "92254225c45b474a9a8da5f25017e440" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 33445 } } peers { permanent_uuid: "e4076bca9109495a9882b31f93df47a3" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 46211 } } }
I20260621 02:55:51.582923 1888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:51.583956 1893 mvcc.cc:205] Tried to move back new op lower bound from 7299115218906853376 to 7299115218531377152. Current Snapshot: MvccSnapshot[applied={T|T < 7299115218906853376}]
I20260621 02:55:51.584340 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:51.485671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.11.109.62:35227)
0621 02:55:51.485841 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.485846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.520222 (+ 34376us) server_negotiation.cc:407] Connection header received
0621 02:55:51.524533 (+ 4311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.524543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.524676 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.524768 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.534153 (+ 9385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.535373 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.548404 (+ 13031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:51.549432 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:51.549576 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:51.568070 (+ 18494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:51.568083 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:51.568088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:51.568226 (+ 138us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:51.579018 (+ 10792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.580230 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.580239 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.580245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.580461 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:51.581035 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:51.581043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:51.581049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:51.581548 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:51.583282 (+ 1734us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:51.583442 (+ 160us) server_negotiation.cc:299] Negotiation successful
0621 02:55:51.583777 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:55:51.587087 1874 catalog_manager.cc:1347] Generated new cluster ID: d67daba5b2644cd0af669bcfbe00fe2e
I20260621 02:55:51.587206 1874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:51.667192 1908 catalog_manager.cc:1259] Loaded cluster ID: d67daba5b2644cd0af669bcfbe00fe2e
I20260621 02:55:51.667426 1908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: loading cluster ID for follower catalog manager: success
W20260621 02:55:51.668746 1908 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260621 02:55:51.668952 1908 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:55:51.680972 1874 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:51.685770 1874 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:51.701247 1874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e025e3d8ab7e4e88aeb090a2b563c1c8: Generated new TSK 0
I20260621 02:55:51.702275 1874 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:55:51.860358 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:51.844357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.11.109.61:33445)
0621 02:55:51.844595 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.844604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.858360 (+ 13756us) server_negotiation.cc:407] Connection header received
0621 02:55:51.858677 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.858684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.858786 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.858870 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.860080 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51304: BlockingRecv error: recv got EOF from 127.0.0.1:51304 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260621 02:55:51.860996 1699 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:51.843723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.11.109.62:35227)
0621 02:55:51.844163 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.844172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.858084 (+ 13912us) server_negotiation.cc:407] Connection header received
0621 02:55:51.859499 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.859505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.859611 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.859683 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.860760 (+ 1077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42314: BlockingRecv error: recv got EOF from 127.0.0.1:42314 (error 108)
Metrics: {"server-negotiator.queue_time_us":229}
W20260621 02:55:51.864823 1847 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:51.858646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.11.109.60:46211)
0621 02:55:51.858837 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.858844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.862796 (+ 3952us) server_negotiation.cc:407] Connection header received
0621 02:55:51.863497 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.863502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.863611 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.863682 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.864599 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56324: BlockingRecv error: recv got EOF from 127.0.0.1:56324 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260621 02:55:51.916328 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:51.914084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.11.109.61:33445)
0621 02:55:51.914260 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.914266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.914289 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:55:51.914604 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.914611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.914772 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.914859 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.915993 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51314: BlockingRecv error: recv got EOF from 127.0.0.1:51314 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260621 02:55:51.958492 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:51.948472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.11.109.61:33445)
0621 02:55:51.948685 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.948691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:51.956524 (+ 7833us) server_negotiation.cc:407] Connection header received
0621 02:55:51.957013 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:51.957019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:51.957113 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:51.957207 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:51.958261 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51316: BlockingRecv error: recv got EOF from 127.0.0.1:51316 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260621 02:55:52.005002 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:51.995544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.11.109.61:33445)
0621 02:55:51.995755 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:51.995761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.003061 (+ 7300us) server_negotiation.cc:407] Connection header received
0621 02:55:52.003516 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.003523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.003653 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.003724 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.004750 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51322: BlockingRecv error: recv got EOF from 127.0.0.1:51322 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 02:55:52.057298 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.047732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.11.109.61:33445)
0621 02:55:52.047914 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.047920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.055404 (+ 7484us) server_negotiation.cc:407] Connection header received
0621 02:55:52.055864 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.055870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.055990 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.056065 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.057034 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51334: BlockingRecv error: recv got EOF from 127.0.0.1:51334 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 02:55:52.071170 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.069015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.11.109.61:33445)
0621 02:55:52.069273 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.069282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.069788 (+ 506us) server_negotiation.cc:407] Connection header received
0621 02:55:52.069998 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.070010 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.070082 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.070192 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.070924 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51350: BlockingRecv error: recv got EOF from 127.0.0.1:51350 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 02:55:52.092682 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.090489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.11.109.61:33445)
0621 02:55:52.090715 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.090725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.091264 (+ 539us) server_negotiation.cc:407] Connection header received
0621 02:55:52.091483 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.091488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.091555 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.091662 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.092415 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51362: BlockingRecv error: recv got EOF from 127.0.0.1:51362 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 02:55:52.130849 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.128486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.11.109.61:33445)
0621 02:55:52.128744 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.128753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.129320 (+ 567us) server_negotiation.cc:407] Connection header received
0621 02:55:52.129564 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.129570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.129702 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.129817 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.130578 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51376: BlockingRecv error: recv got EOF from 127.0.0.1:51376 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260621 02:55:52.200923 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.198517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.11.109.61:33445)
0621 02:55:52.198738 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.198748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.199299 (+ 551us) server_negotiation.cc:407] Connection header received
0621 02:55:52.199625 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.199630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.199706 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.199763 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.200712 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51384: BlockingRecv error: recv got EOF from 127.0.0.1:51384 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:55:52.274852 1861 catalog_manager.cc:1259] Loaded cluster ID: d67daba5b2644cd0af669bcfbe00fe2e
I20260621 02:55:52.274991 1861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: loading cluster ID for follower catalog manager: success
I20260621 02:55:52.277361 1861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: acquiring CA information for follower catalog manager: success
I20260621 02:55:52.279220 1861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4076bca9109495a9882b31f93df47a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:55:52.335887 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.333756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.11.109.61:33445)
0621 02:55:52.333966 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.333978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.334444 (+ 466us) server_negotiation.cc:407] Connection header received
0621 02:55:52.334639 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.334643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.334712 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.334842 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.335686 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51386: BlockingRecv error: recv got EOF from 127.0.0.1:51386 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:55:52.597450 1774 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:52.595485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.11.109.61:33445)
0621 02:55:52.595710 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:52.595718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:52.596068 (+ 350us) server_negotiation.cc:407] Connection header received
0621 02:55:52.596323 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:52.596327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:52.596410 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:52.596519 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:52.597279 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51388: BlockingRecv error: recv got EOF from 127.0.0.1:51388 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:55:52.672350 1908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: acquiring CA information for follower catalog manager: success
I20260621 02:55:52.674176 1908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92254225c45b474a9a8da5f25017e440: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:55:53.118940 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:53.113503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.11.109.61:33445)
0621 02:55:53.113989 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:53.113993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:53.114012 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:55:53.114192 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:53.114196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:53.114337 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:53.114438 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:53.114882 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.115567 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.116872 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.117320 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.117412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:53.117836 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:53.118374 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:53.118479 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:53.118555 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 02:55:53.118615 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":238,"threads_started":1}
I20260621 02:55:53.130301 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:53.124678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.11.109.60:46211)
0621 02:55:53.125169 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:53.125177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:53.125430 (+ 253us) server_negotiation.cc:407] Connection header received
0621 02:55:53.125581 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:53.125586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:53.125728 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:53.125865 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:53.126310 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.127027 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.128299 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.128665 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.128750 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:53.129132 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:53.129665 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:53.129766 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:53.129837 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:55:53.129910 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260621 02:55:53.143705 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:53.136047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.11.109.61:33445)
0621 02:55:53.136278 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:53.136286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:53.138001 (+ 1715us) server_negotiation.cc:407] Connection header received
0621 02:55:53.138218 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:53.138223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:53.138322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:53.138398 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:53.139025 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.139855 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.141742 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.142343 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.142472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:53.142904 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:53.143014 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:53.143122 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:53.143306 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 02:55:53.143381 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:55:53.144433 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:53.136959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.11.109.60:46211)
0621 02:55:53.137348 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:53.137357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:53.138195 (+ 838us) server_negotiation.cc:407] Connection header received
0621 02:55:53.138410 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:53.138417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:53.138503 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:53.138588 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:53.139325 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.140353 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.142325 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.142964 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.143078 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:53.143501 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:53.143638 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:53.143702 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:53.144102 (+ 400us) server_negotiation.cc:299] Negotiation successful
0621 02:55:53.144159 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260621 02:55:53.145355 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:53.135984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.11.109.62:35227)
0621 02:55:53.137350 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:53.137359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:53.137384 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:55:53.137514 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:53.137522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:53.137789 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:53.137960 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:53.138764 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.139727 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.141811 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:53.142422 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:53.142557 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:53.143051 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:53.143462 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:53.143558 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:53.143802 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:55:53.145051 (+ 1249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1093,"threads_started":1}
I20260621 02:55:53.149861 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1629
I20260621 02:55:53.151113 1691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:53.258810 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1629
W20260621 02:55:53.278980 1721 connection.cc:535] server connection from 127.0.0.1:44686 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 02:55:53.279422 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1703
I20260621 02:55:53.280488 1765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:53.383611 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1703
I20260621 02:55:53.403128 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 1777
I20260621 02:55:53.404268 1839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:53.509680 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 1777
2026-06-21T02:55:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5055 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:55:53 dist-test-slave-bc52 krb5kdc[1994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:55:53 dist-test-slave-bc52 krb5kdc[1994](info): set up 2 sockets
Jun 21 02:55:53 dist-test-slave-bc52 krb5kdc[1994](info): commencing operation
krb5kdc: starting...
W20260621 02:55:54.605800 11700 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/test-user.keytab.
Jun 21 02:55:54 dist-test-slave-bc52 krb5kdc[1994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:55:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:55:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
Entry for principal kudu/127.11.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab.
I20260621 02:55:54.842159 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41141
--webserver_interface=127.11.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:41141,127.11.109.61:40325,127.11.109.60:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:55:55.160825 2007 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:55:55.161136 2007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:55.161262 2007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:55.161355 2007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:55.168402 2007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:55.168489 2007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:55.168545 2007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:55.168596 2007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:55.168663 2007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:55.178299 2007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:41141,127.11.109.61:40325,127.11.109.60:36745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.62.keytab
--principal=kudu/127.11.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.62:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:55.179481 2007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:55.180729 2007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:55.191906 2012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:55.191860 2015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:55.191965 2013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:55.192857 2007 server_base.cc:1034] running on GCE node
I20260621 02:55:55.193575 2007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:55.194844 2007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:55.196058 2007 hybrid_clock.cc:648] HybridClock initialized: now 1782010555196003 us; error 34 us; skew 500 ppm
I20260621 02:55:55.196414 2007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:55 dist-test-slave-bc52 krb5kdc[1994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:55.202943 2007 init.cc:377] Logged in from keytab as kudu/127.11.109.62@KRBTEST.COM (short username kudu)
I20260621 02:55:55.204370 2007 webserver.cc:460] Webserver started at http://127.11.109.62:44013/ using document root <none> and password file <none>
I20260621 02:55:55.205062 2007 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:55.205173 2007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:55.205543 2007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:55.208410 2007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data/instance:
uuid: "65cd2599da304eeb81ce8acc28c9df02"
format_stamp: "Formatted at 2026-06-21 02:55:55 on dist-test-slave-bc52"
server_key: "3c428fed9cdccc12385c80f48715e725"
server_key_iv: "0100326d505aa07c2bd71acc28279188"
server_key_version: "encryptionkey@0"
I20260621 02:55:55.209218 2007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal/instance:
uuid: "65cd2599da304eeb81ce8acc28c9df02"
format_stamp: "Formatted at 2026-06-21 02:55:55 on dist-test-slave-bc52"
server_key: "3c428fed9cdccc12385c80f48715e725"
server_key_iv: "0100326d505aa07c2bd71acc28279188"
server_key_version: "encryptionkey@0"
I20260621 02:55:55.214010 2007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:55:55.217146 2022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:55.218420 2007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:55:55.218598 2007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal
uuid: "65cd2599da304eeb81ce8acc28c9df02"
format_stamp: "Formatted at 2026-06-21 02:55:55 on dist-test-slave-bc52"
server_key: "3c428fed9cdccc12385c80f48715e725"
server_key_iv: "0100326d505aa07c2bd71acc28279188"
server_key_version: "encryptionkey@0"
I20260621 02:55:55.218812 2007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:55.243779 2007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:55.247524 2007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:55.247786 2007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:55.265933 2007 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.62:41141
I20260621 02:55:55.265955 2073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.62:41141 every 8 connection(s)
I20260621 02:55:55.267195 2007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/data/info.pb
I20260621 02:55:55.270596 2074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:55.274976 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 2007
I20260621 02:55:55.275167 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-0/wal/instance
I20260621 02:55:55.275707 11700 external_mini_cluster.cc:1428] Setting key 1668a5c7b6f6e6381276aadead3fcd0f
I20260621 02:55:55.277904 2074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:55 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:55.300240 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.281598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41141 (local address 127.0.0.1:47694)
0621 02:55:55.282465 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.282519 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:55.284411 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:55.285220 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:55.285225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:55.285932 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:55.286256 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.286272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.287870 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.287878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.288918 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.288929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.289897 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.289903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.290089 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.290778 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:55.290798 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:55.293391 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:55.296677 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.296685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.296693 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.297055 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.297575 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.297578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.297580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.297727 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.298624 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:55.298630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:55.298797 (+ 167us) client_negotiation.cc:769] Sending connection context
0621 02:55:55.298977 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 02:55:55.299303 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":424,"threads_started":1}
I20260621 02:55:55.300300 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.281598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.11.109.62:41141)
0621 02:55:55.283246 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:55.283258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:55.283303 (+ 45us) server_negotiation.cc:407] Connection header received
0621 02:55:55.284769 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:55.284774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:55.284986 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:55.285218 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:55.286463 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.287722 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.289075 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.289682 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.289872 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.293596 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:55.293603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:55.293620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:55.293652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:55.296485 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.297242 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.297253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.297259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.297465 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.297847 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.297855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.297860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.298488 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:55.298695 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:55.299178 (+ 483us) server_negotiation.cc:299] Negotiation successful
0621 02:55:55.299436 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1436,"thread_start_us":1245,"threads_started":1}
WARNING: no policy specified for kudu/127.11.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.61@KRBTEST.COM" created.
I20260621 02:55:55.303002 2074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:55.304049 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.303576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40436)
0621 02:55:55.303775 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.303891 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 02:55:55.304292 2025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
I20260621 02:55:55.305914 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.305461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40440)
0621 02:55:55.305627 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.305752 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:55:55.306100 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:40325: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab.
Entry for principal kudu/127.11.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab.
I20260621 02:55:55.330641 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:40325
--webserver_interface=127.11.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab
--principal=kudu/127.11.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:41141,127.11.109.61:40325,127.11.109.60:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:55:55.355419 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } attempt: 1
I20260621 02:55:55.356577 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.356151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40448)
0621 02:55:55.356343 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.356445 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 02:55:55.357537 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.357165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40456)
0621 02:55:55.357326 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.357394 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260621 02:55:55.357738 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:40325: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
I20260621 02:55:55.426015 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } attempt: 2
I20260621 02:55:55.427155 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.426688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40470)
0621 02:55:55.426929 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.427016 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260621 02:55:55.428169 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.427813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40474)
0621 02:55:55.427955 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.428036 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 02:55:55.428404 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:40325: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
I20260621 02:55:55.519662 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } attempt: 3
I20260621 02:55:55.520758 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.520348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40480)
0621 02:55:55.520531 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.520618 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:55:55.521754 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.521378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40488)
0621 02:55:55.521514 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.521605 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 02:55:55.521937 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:40325: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
W20260621 02:55:55.648005 2081 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:55:55.648348 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:55.648473 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:55.648605 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:55.655844 2081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:55.655923 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:55.655978 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:55.656019 2081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:55.656100 2081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:55.665208 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } attempt: 4
I20260621 02:55:55.665769 2081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:41141,127.11.109.61:40325,127.11.109.60:36745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.61.keytab
--principal=kudu/127.11.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.61:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:55.666455 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.665823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40496)
0621 02:55:55.666180 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.666292 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260621 02:55:55.667217 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:55.667482 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.667070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40500)
0621 02:55:55.667240 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.667341 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 02:55:55.667721 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.61:40325: Network error: Client connection negotiation failed: client connection to 127.11.109.61:40325: connect: Connection refused (error 111)
I20260621 02:55:55.668620 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:55.680095 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:55.680154 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:55.680186 2089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:55.680910 2081 server_base.cc:1034] running on GCE node
I20260621 02:55:55.681725 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:55.683014 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:55.684258 2081 hybrid_clock.cc:648] HybridClock initialized: now 1782010555684204 us; error 56 us; skew 500 ppm
I20260621 02:55:55.684589 2081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:55:55 dist-test-slave-bc52 krb5kdc[1994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:55.687577 2081 init.cc:377] Logged in from keytab as kudu/127.11.109.61@KRBTEST.COM (short username kudu)
I20260621 02:55:55.688948 2081 webserver.cc:460] Webserver started at http://127.11.109.61:46611/ using document root <none> and password file <none>
I20260621 02:55:55.689666 2081 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:55.689795 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:55.690101 2081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:55.692981 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data/instance:
uuid: "3cf7eebe55c944b392973ec279c94ecb"
format_stamp: "Formatted at 2026-06-21 02:55:55 on dist-test-slave-bc52"
server_key: "143cfe955d99eae65e16a1a2e3e56dfa"
server_key_iv: "65a4e30831a945ce9012586557f915a9"
server_key_version: "encryptionkey@0"
I20260621 02:55:55.693840 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal/instance:
uuid: "3cf7eebe55c944b392973ec279c94ecb"
format_stamp: "Formatted at 2026-06-21 02:55:55 on dist-test-slave-bc52"
server_key: "143cfe955d99eae65e16a1a2e3e56dfa"
server_key_iv: "65a4e30831a945ce9012586557f915a9"
server_key_version: "encryptionkey@0"
I20260621 02:55:55.698494 2081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260621 02:55:55.701843 2096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:55.702977 2081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:55.703166 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal
uuid: "3cf7eebe55c944b392973ec279c94ecb"
format_stamp: "Formatted at 2026-06-21 02:55:55 on dist-test-slave-bc52"
server_key: "143cfe955d99eae65e16a1a2e3e56dfa"
server_key_iv: "65a4e30831a945ce9012586557f915a9"
server_key_version: "encryptionkey@0"
I20260621 02:55:55.703382 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:55.728374 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:55.732128 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:55.732378 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:55.749676 2081 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.61:40325
I20260621 02:55:55.749711 2147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.61:40325 every 8 connection(s)
I20260621 02:55:55.750897 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/data/info.pb
I20260621 02:55:55.752525 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 2081
I20260621 02:55:55.752732 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-1/wal/instance
I20260621 02:55:55.753329 11700 external_mini_cluster.cc:1428] Setting key 3e16d4bf77b3c0cc743c8b88c9cf47d0
I20260621 02:55:55.755000 2148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:55.763978 2148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:55 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.109.60@KRBTEST.COM" created.
I20260621 02:55:55.784008 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.766374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.11.109.62:41141)
0621 02:55:55.766537 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:55.766542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:55.767829 (+ 1287us) server_negotiation.cc:407] Connection header received
0621 02:55:55.769552 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:55.769558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:55.769641 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:55.769747 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:55.771052 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.771749 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.773293 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.773698 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.773844 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.777062 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:55.777066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:55.777069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:55.777100 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:55.780041 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.780992 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.781000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.781002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.781221 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.781804 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.781809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.781813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.782264 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:55.782428 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:55.783491 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0621 02:55:55.783717 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:55:55.785056 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.766999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41141 (local address 127.0.0.1:47704)
0621 02:55:55.767678 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.767712 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:55.769303 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:55.769960 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:55.769968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:55.770436 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:55.770893 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.770907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.771914 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.771918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.773087 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.773098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.773920 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.773927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.774245 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.774870 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:55.774891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:55.776843 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:55.780211 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.780221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.780238 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.780818 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.781321 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.781341 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.781346 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.781676 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.782412 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:55.782451 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:55.782986 (+ 535us) client_negotiation.cc:769] Sending connection context
0621 02:55:55.783352 (+ 366us) client_negotiation.cc:240] Negotiation successful
0621 02:55:55.783797 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":272,"threads_started":1}
I20260621 02:55:55.787113 2148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:55 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:55.801023 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.787712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40510)
0621 02:55:55.787859 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.787878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:55.787973 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:55.788976 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:55.788980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:55.789426 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:55.789542 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.789549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.790887 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.790913 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.791736 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.791745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.792516 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.792522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.792718 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.793316 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:55.793331 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:55.795581 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:55.798656 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.798661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.798664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.798989 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.799358 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.799364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.799368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.799505 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.800071 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:55.800075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:55.800202 (+ 127us) client_negotiation.cc:769] Sending connection context
0621 02:55:55.800355 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 02:55:55.800612 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:55:55.801103 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.788027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.11.109.61:40325)
0621 02:55:55.788610 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:55.788615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:55.788640 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:55:55.788678 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:55.788683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:55.788841 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:55.788985 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:55.789733 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.790761 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.791866 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.792379 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.792513 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.795784 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:55.795794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:55.795805 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:55.795834 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:55.798525 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.799135 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.799141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.799143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.799255 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.799601 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.799608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.799611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.799948 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:55.800167 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:55.800489 (+ 322us) server_negotiation.cc:299] Negotiation successful
0621 02:55:55.800722 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":452,"thread_start_us":332,"threads_started":1}
I20260621 02:55:55.803265 2148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:55.804325 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.803889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:59964)
0621 02:55:55.804058 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.804190 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:55:55.804658 2100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:55.806167 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.805871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:59970)
0621 02:55:55.805998 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.806071 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 02:55:55.806440 2148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab.
Entry for principal kudu/127.11.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab.
I20260621 02:55:55.810075 11700 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
/tmp/dist-test-taskSYTxIs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:36745
--webserver_interface=127.11.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab
--principal=kudu/127.11.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.109.20:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.109.62:41141,127.11.109.61:40325,127.11.109.60:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:55:55.855834 2148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 1
I20260621 02:55:55.857076 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.856629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:59978)
0621 02:55:55.856830 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.856932 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 02:55:55.858229 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.857914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:59980)
0621 02:55:55.858072 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.858137 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:55:55.858549 2148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:55.926810 2148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 2
I20260621 02:55:55.928068 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.927475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:59992)
0621 02:55:55.927713 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.927837 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:55:55.929200 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.928778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60006)
0621 02:55:55.928937 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.929023 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:55:55.929428 2148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:55.967085 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } attempt: 5
Jun 21 02:55:55 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:55.980405 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.968068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.11.109.61:40325)
0621 02:55:55.968315 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:55.968319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:55.968334 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:55:55.968384 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:55.968388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:55.968444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:55.968550 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:55.969397 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.970078 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.971227 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.971660 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.971795 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.975257 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:55.975262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:55.975265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:55.975296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:55.977909 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.978570 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.978581 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.978584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.978696 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:55.979057 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:55.979062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:55.979065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:55.979317 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:55.979484 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:55.979885 (+ 401us) server_negotiation.cc:299] Negotiation successful
0621 02:55:55.980076 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:55:55.980405 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.967809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40520)
0621 02:55:55.968002 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.968028 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:55.968160 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:55.968596 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:55.968602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:55.969002 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:55.969239 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.969254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.970215 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.970224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.971070 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:55.971081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:55.971805 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:55.971810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:55.972016 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:55.972689 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:55.972706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:55.975064 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:55.978053 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.978059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.978063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.978434 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.978805 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:55.978810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:55.978814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:55.978940 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:55.979453 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:55.979458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:55.979606 (+ 148us) client_negotiation.cc:769] Sending connection context
0621 02:55:55.979784 (+ 178us) client_negotiation.cc:240] Negotiation successful
0621 02:55:55.980046 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:55:55.981609 2074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:55.982892 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.982354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60020)
0621 02:55:55.982584 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.982692 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 02:55:55.983860 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:55.983500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60028)
0621 02:55:55.983644 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:55.983742 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:55:55.984118 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:56.020663 2148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 3
I20260621 02:55:56.022001 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.021388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60042)
0621 02:55:56.021664 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.021794 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 02:55:56.023159 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.022687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60048)
0621 02:55:56.022902 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.022975 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260621 02:55:56.023435 2148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:56.035348 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 1
I20260621 02:55:56.036384 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.035953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60060)
0621 02:55:56.036142 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.036222 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 02:55:56.037463 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.037127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60076)
0621 02:55:56.037297 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.037361 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260621 02:55:56.037742 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:56.105952 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 2
I20260621 02:55:56.107257 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.106715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60092)
0621 02:55:56.106965 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.107051 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:55:56.108374 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.107913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60100)
0621 02:55:56.108130 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.108222 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260621 02:55:56.108579 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
W20260621 02:55:56.132963 2155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:55:56.133261 2155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:55:56.133328 2155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:55:56.133490 2155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:55:56.140339 2155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:55:56.140412 2155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:55:56.140477 2155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:55:56.140523 2155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:55:56.140652 2155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:55:56.149957 2155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.109.20:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.109.62:41141,127.11.109.61:40325,127.11.109.60:36745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/krb5kdc/kudu_127.11.109.60.keytab
--principal=kudu/127.11.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.109.60:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:55:56.151240 2155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:55:56.152534 2155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:55:56.163990 2160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:56.164068 2163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:55:56.164095 2161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:55:56.164829 2155 server_base.cc:1034] running on GCE node
I20260621 02:55:56.165648 2155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:55:56.166637 2148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 4
I20260621 02:55:56.166944 2155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:55:56.167764 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.167242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60102)
0621 02:55:56.167476 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.167572 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:55:56.168166 2155 hybrid_clock.cc:648] HybridClock initialized: now 1782010556168094 us; error 50 us; skew 500 ppm
I20260621 02:55:56.168469 2155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:55:56.168725 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.168390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60118)
0621 02:55:56.168536 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.168631 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:55:56.168963 2148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010556, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:55:56.171675 2155 init.cc:377] Logged in from keytab as kudu/127.11.109.60@KRBTEST.COM (short username kudu)
I20260621 02:55:56.173018 2155 webserver.cc:460] Webserver started at http://127.11.109.60:44653/ using document root <none> and password file <none>
I20260621 02:55:56.173696 2155 fs_manager.cc:359] Metadata directory not provided
I20260621 02:55:56.173780 2155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:55:56.174027 2155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:55:56.176803 2155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data/instance:
uuid: "8273173e01d249d8a70bb6fb9c159224"
format_stamp: "Formatted at 2026-06-21 02:55:56 on dist-test-slave-bc52"
server_key: "84a81aeb8f8bb616697ee32a4ea5717c"
server_key_iv: "bb8ce8589bdb666f0821df70e38882c7"
server_key_version: "encryptionkey@0"
I20260621 02:55:56.177623 2155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal/instance:
uuid: "8273173e01d249d8a70bb6fb9c159224"
format_stamp: "Formatted at 2026-06-21 02:55:56 on dist-test-slave-bc52"
server_key: "84a81aeb8f8bb616697ee32a4ea5717c"
server_key_iv: "bb8ce8589bdb666f0821df70e38882c7"
server_key_version: "encryptionkey@0"
I20260621 02:55:56.182317 2155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:55:56.185472 2170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:55:56.186720 2155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:55:56.186846 2155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data,/tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal
uuid: "8273173e01d249d8a70bb6fb9c159224"
format_stamp: "Formatted at 2026-06-21 02:55:56 on dist-test-slave-bc52"
server_key: "84a81aeb8f8bb616697ee32a4ea5717c"
server_key_iv: "bb8ce8589bdb666f0821df70e38882c7"
server_key_version: "encryptionkey@0"
I20260621 02:55:56.187003 2155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:55:56.209120 2155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:55:56.213276 2155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:55:56.213480 2155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:55:56.214799 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 3
I20260621 02:55:56.215922 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.215371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60130)
0621 02:55:56.215603 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.215726 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 02:55:56.216989 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.216541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60138)
0621 02:55:56.216761 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.216849 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260621 02:55:56.217259 2074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.109.60:36745: Network error: Client connection negotiation failed: client connection to 127.11.109.60:36745: connect: Connection refused (error 111)
I20260621 02:55:56.231477 2155 rpc_server.cc:307] RPC server started. Bound to: 127.11.109.60:36745
I20260621 02:55:56.231498 2221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.109.60:36745 every 8 connection(s)
I20260621 02:55:56.232661 2155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/data/info.pb
I20260621 02:55:56.236024 2222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:55:56.242686 11700 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu as pid 2155
I20260621 02:55:56.242859 11700 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSYTxIs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782010204918124-11700-0/minicluster-data/master-2/wal/instance
I20260621 02:55:56.242133 2222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:56.243402 11700 external_mini_cluster.cc:1428] Setting key ae8230c1a5a19c3c4354c900648f5b56
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010556, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:55:56.262101 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.244930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.11.109.62:41141)
0621 02:55:56.245117 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.245123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.246198 (+ 1075us) server_negotiation.cc:407] Connection header received
0621 02:55:56.247731 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.247740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.247830 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.247932 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.249702 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.250390 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.252017 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.252716 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.252840 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.256161 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.256170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.256176 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.256224 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.258985 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.259753 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.259758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.259762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.259920 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.260299 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.260306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.260310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.260622 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.260803 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.261464 (+ 661us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.261731 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:55:56.263129 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.245123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.62:41141 (local address 127.0.0.1:47714)
0621 02:55:56.246033 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.246076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:56.247492 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:56.248368 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:56.248372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:56.248933 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:56.249528 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.249546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.250534 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.250539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.251864 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.251893 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.252947 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.252953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.253066 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.253716 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:56.253736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:56.255961 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:56.259118 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.259129 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.259139 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.259616 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.260026 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.260029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.260031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.260174 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.260749 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:56.260754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:56.261064 (+ 310us) client_negotiation.cc:769] Sending connection context
0621 02:55:56.261368 (+ 304us) client_negotiation.cc:240] Negotiation successful
0621 02:55:56.261825 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":3,"thread_start_us":434,"threads_started":1}
I20260621 02:55:56.263150 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.245121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.11.109.60:36745)
0621 02:55:56.245942 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.245951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.245989 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:55:56.247477 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.247495 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.247782 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.247959 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.249169 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.250757 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.251930 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.252440 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.252676 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.256883 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.256889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.256898 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.256954 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.259810 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.260477 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.260482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.260484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.260602 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.260999 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.261004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.261007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.261434 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.261721 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.262054 (+ 333us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.262498 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":614,"spinlock_wait_cycles":15232,"thread_start_us":385,"threads_started":1}
I20260621 02:55:56.265437 2222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010556, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.62@KRBTEST.COM
I20260621 02:55:56.279070 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.266187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.11.109.61:40325)
0621 02:55:56.266388 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.266397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.266418 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:55:56.266478 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.266483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.266571 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.266664 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.267422 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.268133 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.269332 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.269754 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.269890 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.273241 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.273249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.273254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.273298 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.276610 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.277234 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.277242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.277247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.277390 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.277886 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.277890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.277892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.278134 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.278303 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.278686 (+ 383us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.278835 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:55:56.279232 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.266067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.61:40325 (local address 127.0.0.1:40536)
0621 02:55:56.266231 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.266249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:56.266340 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:56.266702 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:56.266709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:56.267134 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:56.267270 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.267278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.268269 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.268277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.269144 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.269154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.269897 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.269903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.270103 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.270659 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:56.270675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:56.273017 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:56.276738 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.276741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.276743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.277084 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.277561 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.277566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.277570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.277754 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.278272 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:56.278276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:56.278447 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 02:55:56.278600 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 02:55:56.278868 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:55:56.280236 2222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:55:56.282651 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.265769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.11.109.62:41141)
0621 02:55:56.266332 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.266337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.266573 (+ 236us) server_negotiation.cc:407] Connection header received
0621 02:55:56.266761 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.266769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.266842 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.266940 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.268363 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.269476 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.271195 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.271886 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.272017 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.276756 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.276762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.276766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.276800 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.279974 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.280633 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.280638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.280641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.280792 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.281338 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.281346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.281351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.281723 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.281928 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.282284 (+ 356us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.282433 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448}
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010556, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.60@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.109.61@KRBTEST.COM
I20260621 02:55:56.294281 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.280987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60154)
0621 02:55:56.281496 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.281514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:56.281604 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:56.282013 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:56.282017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:56.282464 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:56.282627 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.282648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.283894 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.283898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.285019 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.285030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.286058 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.286065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.286273 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.286794 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:56.286814 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:56.288815 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:56.291889 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.291896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.291900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.292230 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.292606 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.292613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.292617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.292793 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.293301 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:56.293306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:56.293456 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 02:55:56.293594 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:55:56.293877 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 02:55:56.294342 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.281412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.11.109.60:36745)
0621 02:55:56.281614 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.281619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.281669 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:55:56.281710 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.281714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.281795 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.281879 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.282798 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.283772 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.285153 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.285870 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.286010 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.288978 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.288983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.288986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.289009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.291779 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.292377 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.292380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.292382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.292483 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.292904 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.292907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.292909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.293136 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.293334 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.293679 (+ 345us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.294013 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":67}
I20260621 02:55:56.299065 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.284210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.11.109.61:40325)
0621 02:55:56.284404 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.284409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.284428 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:55:56.284473 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.284477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.284563 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.284654 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.286029 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.287087 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.288470 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.288887 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.289015 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.293269 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.293275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.293279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.293319 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.296606 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.297204 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.297209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.297214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.297345 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.297694 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.297702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.297707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.298009 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.298286 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.298513 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.298739 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:55:56.300084 2222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: Bootstrap starting.
I20260621 02:55:56.303519 11700 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:55:56.303723 2222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:56.304883 2222 log.cc:824] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:56.307878 2222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: No bootstrap required, opened a new log
I20260621 02:55:56.306545 2049 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "65cd2599da304eeb81ce8acc28c9df02"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47730
I20260621 02:55:56.314651 2222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.314945 2222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8273173e01d249d8a70bb6fb9c159224, State: Initialized, Role: FOLLOWER
I20260621 02:55:56.315398 2222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.316196 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.316500 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.317420 2222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:55:56.326699 2239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:55:56.326870 2239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:55:56.324435 2155 master_runner.cc:186] Error getting master registration for 127.11.109.62:41141: OK, instance_id { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" instance_seqno: 1782010555260801 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:55:56.332404 2155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:55:56.394421 2074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 4
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.62@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:55:56.409617 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.395169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60162)
0621 02:55:56.395360 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.395381 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:56.395512 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:56.396106 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:56.396110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:56.396623 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:56.396823 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.396833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.397936 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.397941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.398871 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.398883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.399619 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.399624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.399830 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.400442 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:56.400460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:56.402727 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:56.406842 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.406850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.406855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.407337 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.407751 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.407755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.407759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.407939 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.408612 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:56.408617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:56.408791 (+ 174us) client_negotiation.cc:769] Sending connection context
0621 02:55:56.408933 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:55:56.409224 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 02:55:56.409828 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.395245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.11.109.60:36745)
0621 02:55:56.395421 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.395429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.395452 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:55:56.395728 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.395737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.395832 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.395919 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.396964 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.397783 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.399026 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.399468 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.399609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.402953 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.402959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.402965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.403002 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.406693 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.407459 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.407464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.407470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.407628 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.408044 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.408049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.408053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.408486 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.408655 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.409059 (+ 404us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.409311 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:55:56.417403 2074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02: Bootstrap starting.
I20260621 02:55:56.425138 2074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:56.426662 2074 log.cc:824] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:56.429996 2074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02: No bootstrap required, opened a new log
I20260621 02:55:56.437677 2074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.438077 2074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cd2599da304eeb81ce8acc28c9df02, State: Initialized, Role: FOLLOWER
I20260621 02:55:56.438546 2074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.439792 2242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.440383 2242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.440907 2074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:56.441025 2049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 02:55:56.441228 2049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:56.445389 2049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.447700 2049 leader_election.cc:288] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [CANDIDATE]: Term 1 election: Requested vote from peers 3cf7eebe55c944b392973ec279c94ecb (127.11.109.61:40325), 8273173e01d249d8a70bb6fb9c159224 (127.11.109.60:36745)
I20260621 02:55:56.448668 2197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65cd2599da304eeb81ce8acc28c9df02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8273173e01d249d8a70bb6fb9c159224"
I20260621 02:55:56.449162 2197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:56.453886 2197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65cd2599da304eeb81ce8acc28c9df02 in term 1.
I20260621 02:55:56.454941 2026 leader_election.cc:302] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65cd2599da304eeb81ce8acc28c9df02, 8273173e01d249d8a70bb6fb9c159224; no voters:
I20260621 02:55:56.455410 2242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 1 FOLLOWER]: Leader election won for term 1
W20260621 02:55:56.465407 2176 tablet.cc:2369] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:55:56.467911 2242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [term 1 LEADER]: Becoming Leader. State: Replica: 65cd2599da304eeb81ce8acc28c9df02, State: Running, Role: LEADER
I20260621 02:55:56.468163 2148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } attempt: 5
I20260621 02:55:56.468638 2242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
Jun 21 02:55:56 dist-test-slave-bc52 krb5kdc[1994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010555, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.109.61@KRBTEST.COM for kudu/127.11.109.60@KRBTEST.COM
I20260621 02:55:56.483110 2246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65cd2599da304eeb81ce8acc28c9df02. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.483891 2246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:56.488566 2255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:55:56.493273 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.468858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.109.60:36745 (local address 127.0.0.1:60168)
0621 02:55:56.469162 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0621 02:55:56.469204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:55:56.469350 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:55:56.469987 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:55:56.469992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:55:56.470544 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:55:56.470820 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.470832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.472471 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.472476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.474046 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:55:56.474060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.475517 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.475549 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:55:56.475977 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.476914 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:55:56.476967 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:55:56.480882 (+ 3915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:55:56.487017 (+ 6135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.487025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.487032 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.487959 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.488778 (+ 819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:55:56.488783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:55:56.488786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:55:56.489029 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:55:56.490138 (+ 1109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:55:56.490143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:55:56.490387 (+ 244us) client_negotiation.cc:769] Sending connection context
0621 02:55:56.490618 (+ 231us) client_negotiation.cc:240] Negotiation successful
0621 02:55:56.491726 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260621 02:55:56.493870 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.468931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.11.109.60:36745)
0621 02:55:56.469120 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.469127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.469278 (+ 151us) server_negotiation.cc:407] Connection header received
0621 02:55:56.469481 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.469486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.469589 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.469700 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.471044 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.472280 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.474278 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.475137 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.475259 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.481202 (+ 5943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.481211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.481218 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.481266 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.486757 (+ 5491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.488232 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.488267 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.488271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.488592 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.489284 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.489294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.489299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.489884 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.491168 (+ 1284us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.491694 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.492494 (+ 800us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:56.500245 2148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: Bootstrap starting.
W20260621 02:55:56.500464 2028 tablet.cc:2369] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:55:56.506665 2148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: Neither blocks nor log segments found. Creating new log.
I20260621 02:55:56.508056 2148 log.cc:824] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: Log is configured to *not* fsync() on all Append() calls
I20260621 02:55:56.518922 2255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:55:56.522778 2148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: No bootstrap required, opened a new log
I20260621 02:55:56.532124 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.479841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.11.109.60:36745)
0621 02:55:56.489112 (+ 9271us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.489119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.495763 (+ 6644us) server_negotiation.cc:407] Connection header received
0621 02:55:56.495952 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.495957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.496032 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.496135 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.502496 (+ 6361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.503697 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.505850 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.506549 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.506658 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.513313 (+ 6655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.513319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.513324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.513372 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.516871 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.521814 (+ 4943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.521823 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.521829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.522028 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.523359 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.523367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.523373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.523796 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.524097 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.531362 (+ 7265us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.531666 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9123,"thread_start_us":9843,"threads_started":1}
I20260621 02:55:56.541882 2148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.542359 2148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf7eebe55c944b392973ec279c94ecb, State: Initialized, Role: FOLLOWER
I20260621 02:55:56.542945 2148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } }
I20260621 02:55:56.543418 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.504499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.11.109.61:40325)
0621 02:55:56.504661 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.504667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.506028 (+ 1361us) server_negotiation.cc:407] Connection header received
0621 02:55:56.506429 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.506437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.506537 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.506608 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.513787 (+ 7179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.514835 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.517099 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.518000 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.518124 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.531495 (+ 13371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.531502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.531507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.531548 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.535472 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.536407 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.536414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.536417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.536661 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.541070 (+ 4409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.541076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.541100 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.541684 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.542178 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.542776 (+ 598us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.543026 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:55:56.547045 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.478562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.11.109.62:41141)
0621 02:55:56.478762 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.478770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.491565 (+ 12795us) server_negotiation.cc:407] Connection header received
0621 02:55:56.491834 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.491840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.491925 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.491997 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.504767 (+ 12770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.510694 (+ 5927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.513725 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:56.515624 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:56.515753 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:56.531293 (+ 15540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:55:56.531301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:55:56.531307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:55:56.536915 (+ 5608us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:55:56.540832 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.543654 (+ 2822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.543662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.543666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.543838 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:55:56.544485 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:55:56.544491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:55:56.544494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:55:56.544845 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:55:56.545095 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:56.545993 (+ 898us) server_negotiation.cc:299] Negotiation successful
0621 02:55:56.546387 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:55:56.566674 2148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:55:56.566751 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65cd2599da304eeb81ce8acc28c9df02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3cf7eebe55c944b392973ec279c94ecb"
I20260621 02:55:56.572911 2123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:55:56.574103 2197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [term 1 FOLLOWER]: Refusing update from remote peer 65cd2599da304eeb81ce8acc28c9df02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:56.575129 2242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:56.576313 2123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65cd2599da304eeb81ce8acc28c9df02 in term 1.
I20260621 02:55:56.577607 2263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.577961 2263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.593219 2123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [term 1 FOLLOWER]: Refusing update from remote peer 65cd2599da304eeb81ce8acc28c9df02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:55:56.594076 2242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:55:56.607487 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65cd2599da304eeb81ce8acc28c9df02. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.607856 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.617923 2263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65cd2599da304eeb81ce8acc28c9df02. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.618254 2263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.619664 2274 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:55:56.623726 2271 mvcc.cc:205] Tried to move back new op lower bound from 7299115239707963392 to 7299115239310090240. Current Snapshot: MvccSnapshot[applied={T|T < 7299115239707963392}]
I20260621 02:55:56.624161 2265 mvcc.cc:205] Tried to move back new op lower bound from 7299115239707963392 to 7299115239310090240. Current Snapshot: MvccSnapshot[applied={T|T < 7299115239707963392}]
I20260621 02:55:56.624869 2263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.625108 2263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.632988 2267 mvcc.cc:205] Tried to move back new op lower bound from 7299115239707963392 to 7299115239310090240. Current Snapshot: MvccSnapshot[applied={T|T < 7299115239707963392}]
I20260621 02:55:56.636283 2228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.636560 2228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:55:56.639894 2242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.640293 2242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:56.640563 2242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65cd2599da304eeb81ce8acc28c9df02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd2599da304eeb81ce8acc28c9df02" member_type: VOTER last_known_addr { host: "127.11.109.62" port: 41141 } } peers { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" member_type: VOTER last_known_addr { host: "127.11.109.61" port: 40325 } } peers { permanent_uuid: "8273173e01d249d8a70bb6fb9c159224" member_type: VOTER last_known_addr { host: "127.11.109.60" port: 36745 } } }
I20260621 02:55:56.640828 2242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02 [sys.catalog]: This master's current role is: LEADER
I20260621 02:55:56.642164 2255 catalog_manager.cc:1347] Generated new cluster ID: ee477a9b2c73419db9d88d98a8472f3f
I20260621 02:55:56.642302 2255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:55:56.653527 2007 master_runner.cc:186] Error getting master registration for 127.11.109.61:40325: OK, instance_id { permanent_uuid: "3cf7eebe55c944b392973ec279c94ecb" instance_seqno: 1782010555744771 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:55:56.694824 2007 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:55:56.771415 2255 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:55:56.773756 2255 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:55:56.810803 2286 catalog_manager.cc:1259] Loaded cluster ID: ee477a9b2c73419db9d88d98a8472f3f
I20260621 02:55:56.811022 2286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: loading cluster ID for follower catalog manager: success
I20260621 02:55:56.814607 2286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: acquiring CA information for follower catalog manager: success
I20260621 02:55:56.818279 2255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65cd2599da304eeb81ce8acc28c9df02: Generated new TSK 0
I20260621 02:55:56.819412 2255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:55:56.835654 2286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3cf7eebe55c944b392973ec279c94ecb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:55:56.953997 2077 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:56.937707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.11.109.62:41141)
0621 02:55:56.937925 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.937931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.951602 (+ 13671us) server_negotiation.cc:407] Connection header received
0621 02:55:56.952282 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.952289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.952393 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.952481 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.953733 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47744: BlockingRecv error: recv got EOF from 127.0.0.1:47744 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:55:56.956187 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:56.941501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.11.109.60:36745)
0621 02:55:56.941720 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.941726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.953861 (+ 12135us) server_negotiation.cc:407] Connection header received
0621 02:55:56.954392 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.954401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.954531 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.954613 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.955865 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60186: BlockingRecv error: recv got EOF from 127.0.0.1:60186 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260621 02:55:56.956831 2152 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:56.950031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.11.109.61:40325)
0621 02:55:56.950211 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.950217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.954897 (+ 4680us) server_negotiation.cc:407] Connection header received
0621 02:55:56.955546 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.955551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.955636 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.955699 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:56.956593 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40560: BlockingRecv error: recv got EOF from 127.0.0.1:40560 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:55:57.006734 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:56.989705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.11.109.61:40325)
0621 02:55:56.989874 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:56.989880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:56.997202 (+ 7322us) server_negotiation.cc:407] Connection header received
0621 02:55:56.997637 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:56.997645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:56.997942 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:56.998112 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:56.998875 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.000171 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.003005 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.003842 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.003962 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.004847 (+ 885us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.005732 (+ 885us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.005862 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.006133 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.006255 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260621 02:55:57.040066 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.037788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.11.109.60:36745)
0621 02:55:57.037985 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.037992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.038636 (+ 644us) server_negotiation.cc:407] Connection header received
0621 02:55:57.038891 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.038895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.038992 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.039116 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.039872 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60188: BlockingRecv error: recv got EOF from 127.0.0.1:60188 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:55:57.054992 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.044761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.11.109.61:40325)
0621 02:55:57.044938 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.044944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.048880 (+ 3936us) server_negotiation.cc:407] Connection header received
0621 02:55:57.049069 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.049076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.049203 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.049305 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.049818 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.050934 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.052931 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.053619 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.053754 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.054198 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.054365 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.054460 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.054616 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.054703 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 02:55:57.067160 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.062110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.11.109.60:36745)
0621 02:55:57.062283 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.062289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.065743 (+ 3454us) server_negotiation.cc:407] Connection header received
0621 02:55:57.065926 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.065932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.066025 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.066129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.066935 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60204: BlockingRecv error: recv got EOF from 127.0.0.1:60204 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:55:57.092406 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.074821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.11.109.61:40325)
0621 02:55:57.075008 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.075013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.085857 (+ 10844us) server_negotiation.cc:407] Connection header received
0621 02:55:57.086053 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.086059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.086156 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.086275 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.086789 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.087994 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.090146 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.090876 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.091014 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.091564 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.091696 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.091787 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.091949 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.092042 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 02:55:57.102012 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.099283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.11.109.60:36745)
0621 02:55:57.099481 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.099488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.100423 (+ 935us) server_negotiation.cc:407] Connection header received
0621 02:55:57.100612 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.100620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.100728 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.100837 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.101789 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60220: BlockingRecv error: recv got EOF from 127.0.0.1:60220 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:55:57.117888 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.110992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.11.109.61:40325)
0621 02:55:57.111217 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.111223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.111703 (+ 480us) server_negotiation.cc:407] Connection header received
0621 02:55:57.111880 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.111887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.111993 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.112097 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.112540 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.113761 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.115723 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.116439 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.116570 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.117022 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.117225 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.117323 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.117514 (+ 191us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.117595 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260621 02:55:57.127079 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.124633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.11.109.60:36745)
0621 02:55:57.124863 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.124869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.125306 (+ 437us) server_negotiation.cc:407] Connection header received
0621 02:55:57.125743 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.125751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.125849 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.125960 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.126861 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60236: BlockingRecv error: recv got EOF from 127.0.0.1:60236 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260621 02:55:57.145128 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.138224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.11.109.61:40325)
0621 02:55:57.138409 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.138415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.138912 (+ 497us) server_negotiation.cc:407] Connection header received
0621 02:55:57.139111 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.139117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.139223 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.139323 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.139763 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.140957 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.142980 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.143689 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.143828 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.144292 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.144449 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.144555 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.144718 (+ 163us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.144804 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:55:57.153847 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.151564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.11.109.60:36745)
0621 02:55:57.151741 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.151747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.152289 (+ 542us) server_negotiation.cc:407] Connection header received
0621 02:55:57.152462 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.152468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.152571 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.152702 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.153581 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60246: BlockingRecv error: recv got EOF from 127.0.0.1:60246 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:57.179855 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.173057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.11.109.61:40325)
0621 02:55:57.173267 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.173274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.173827 (+ 553us) server_negotiation.cc:407] Connection header received
0621 02:55:57.174014 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.174023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.174120 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.174226 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.174722 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.175930 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.177817 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.178479 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.178621 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.179078 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.179215 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.179316 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.179463 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.179552 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 02:55:57.193156 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.190544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.11.109.60:36745)
0621 02:55:57.190766 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.190773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.191500 (+ 727us) server_negotiation.cc:407] Connection header received
0621 02:55:57.191770 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.191776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.191863 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.191967 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.192947 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60248: BlockingRecv error: recv got EOF from 127.0.0.1:60248 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:55:57.239681 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.230046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.11.109.61:40325)
0621 02:55:57.230243 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.230249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.230784 (+ 535us) server_negotiation.cc:407] Connection header received
0621 02:55:57.233536 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.233542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.233660 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.233723 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.234350 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.235514 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.237454 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.238173 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.238267 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.238839 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.238963 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.239026 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.239239 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.239330 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260621 02:55:57.266920 2253 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:55:57.264717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.11.109.60:36745)
0621 02:55:57.264946 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.264952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.265336 (+ 384us) server_negotiation.cc:407] Connection header received
0621 02:55:57.265576 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.265584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.265677 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.265789 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:55:57.266694 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60254: BlockingRecv error: recv got EOF from 127.0.0.1:60254 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 02:55:57.328804 2239 catalog_manager.cc:1259] Loaded cluster ID: ee477a9b2c73419db9d88d98a8472f3f
I20260621 02:55:57.328929 2239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: loading cluster ID for follower catalog manager: success
I20260621 02:55:57.331674 2239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: acquiring CA information for follower catalog manager: success
I20260621 02:55:57.333911 2239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8273173e01d249d8a70bb6fb9c159224: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:55:57.340493 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.334596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.11.109.61:40325)
0621 02:55:57.334837 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.334847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.335417 (+ 570us) server_negotiation.cc:407] Connection header received
0621 02:55:57.335737 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.335743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.335838 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.335936 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.336532 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.337537 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.338856 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.339271 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.339401 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.339798 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.339893 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.339997 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.340142 (+ 145us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.340219 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:55:57.352376 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.346235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.11.109.60:36745)
0621 02:55:57.346420 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.346427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.346914 (+ 487us) server_negotiation.cc:407] Connection header received
0621 02:55:57.347127 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.347131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.347259 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.347350 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.347735 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.348517 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.350395 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.350766 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.350853 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.351358 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.351856 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.351962 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.352073 (+ 111us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.352133 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:55:57.365998 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.358257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.11.109.62:41141)
0621 02:55:57.358454 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.358459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.359008 (+ 549us) server_negotiation.cc:407] Connection header received
0621 02:55:57.359313 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.359319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.359508 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.359636 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.360622 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.361586 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.363514 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.364118 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.364205 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.364862 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.365280 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.365351 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.365603 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.365696 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:55:57.372581 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 2007
I20260621 02:55:57.373605 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.358355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.11.109.60:36745)
0621 02:55:57.358563 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.358571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.366857 (+ 8286us) server_negotiation.cc:407] Connection header received
0621 02:55:57.367120 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.367126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.367219 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.367277 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.367791 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.368893 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.371500 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.372108 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.372222 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.372793 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.372945 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.373026 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.373266 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.373340 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:55:57.373612 2069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:57.374924 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:55:57.359296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.11.109.61:40325)
0621 02:55:57.359504 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:55:57.359512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:55:57.368637 (+ 9125us) server_negotiation.cc:407] Connection header received
0621 02:55:57.368710 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:55:57.368714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:55:57.368809 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:55:57.368938 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:55:57.369673 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.370699 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.372887 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:55:57.373575 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:55:57.373683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:55:57.374208 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:55:57.374344 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:55:57.374438 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:55:57.374579 (+ 141us) server_negotiation.cc:299] Negotiation successful
0621 02:55:57.374649 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:55:57.484611 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 2007
I20260621 02:55:57.506138 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 2081
I20260621 02:55:57.507282 2143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:57.610558 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 2081
I20260621 02:55:57.630717 11700 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu pid 2155
I20260621 02:55:57.631819 2217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:55:57.735482 11700 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSYTxIs/build/asan/bin/kudu with pid 2155
2026-06-21T02:55:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4222 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9277 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (352785 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST